Jul 16 21:47:58.044315 ip-10-0-143-100 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:47:58.044330 ip-10-0-143-100 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:47:58.044341 ip-10-0-143-100 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:47:58.044685 ip-10-0-143-100 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:48:09.571803 ip-10-0-143-100 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:48:09.571822 ip-10-0-143-100 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot abcbf896a0fc436d9edccb8835574306 --
Jul 16 21:49:59.801162 ip-10-0-143-100 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:50:00.134162 ip-10-0-143-100 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.134162 ip-10-0-143-100 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:50:00.134162 ip-10-0-143-100 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.134162 ip-10-0-143-100 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:50:00.134162 ip-10-0-143-100 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.135603 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.135486    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:50:00.141760 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141735    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.141760 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141758    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.141760 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141761    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141764    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141768    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141770    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141773    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141776    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141780    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141783    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141786    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141788    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141791    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141794    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141796    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141799    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141802    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141804    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141807    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141809    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141823    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141826    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.141873 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141828    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141831    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141834    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141837    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141840    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141842    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141845    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141847    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141850    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141853    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141855    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141858    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141862    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141864    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141867    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141870    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141872    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141875    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141878    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.142356 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141885    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141887    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141890    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141893    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141897    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141901    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141904    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141906    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141909    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141911    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141914    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141916    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141919    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141921    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141924    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141927    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141930    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141934    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141937    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141939    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141942    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.143060 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141944    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141947    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141949    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141952    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141967    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141971    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141974    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141977    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141980    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141983    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141986    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141988    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141991    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141994    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.141998    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142000    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142003    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142007    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142010    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.143696 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142013    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142015    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142018    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142021    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142023    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142026    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142028    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142031    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142034    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142037    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142040    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142042    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142045    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142047    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142050    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142052    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142055    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.142058    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144058    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.144166 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144072    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144075    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144079    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144081    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144084    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144087    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144089    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144093    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144095    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144098    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144100    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144103    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144106    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144108    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144111    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144114    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144119    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144121    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144124    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.144615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144127    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144129    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144131    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144134    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144137    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144139    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144142    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144144    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144151    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144154    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144158    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144161    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144164    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144167    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144170    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144172    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144175    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144178    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144181    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144184    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.145104 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144186    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144188    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144191    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144193    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144196    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144198    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144201    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144204    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144206    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144208    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144212    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144214    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144217    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144220    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144223    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144227    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144230    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144234    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144236    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.145591 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144239    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144242    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144244    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144246    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144249    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144252    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144254    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144257    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144259    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144262    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144265    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144267    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144269    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144272    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144274    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144278    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144280    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144282    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144285    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144287    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.146082 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144290    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144292    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144294    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144298    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144302    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144304    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144307    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144309    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144311    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144314    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144317    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144319    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144322    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144324    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144327    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144331    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144334    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144337    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144340    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.146588 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144342    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144746    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144750    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144753    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144755    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144758    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144761    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144763    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144765    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144768    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144771    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144773    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144776    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144778    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144781    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144785    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144789    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144792    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144795    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.147066 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144798    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144800    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144804    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144807    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144811    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144832    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144836    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144840    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144844    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144848    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144850    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144853    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144856    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144858    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144862    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144866    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144869    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144872    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144875    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144877    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.147531 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144880    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144883    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144885    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144888    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144890    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144893    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144895    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144897    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144900    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144902    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144905    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144908    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144910    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144913    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144915    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144917    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144920    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144923    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144925    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.148048 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144927    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144930    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144933    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144935    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144938    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144940    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144943    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144945    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144947    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144950    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144952    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144955    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144957    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144960    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144962    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144965    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144968    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144970    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144973    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144975    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.148493 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144977    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144980    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144983    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144986    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144988    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144991    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144994    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144996    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.144998    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145001    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145004    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145006    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145008    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145011    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145015    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145018    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145020    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145023    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145025    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145027    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.148982 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145030    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145033    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145106    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145117    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145124    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145128    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145133    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145136    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145141    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145145    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145148    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145151    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145154    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145158    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145161    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145164    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145167    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145169    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145173    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145176    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145181    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145184    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145187    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145190    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:50:00.149500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145194    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145197    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145200    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145204    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145207    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145210    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145213    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145215    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145218    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145221    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145226    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145228    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145231    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145234    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145237    2566 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145240    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145244    2566 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145247    2566 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145250    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145253    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145256    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145259    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145262    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145265    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145268    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:50:00.150086 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145271    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145273    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145277    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145280    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145285    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145287    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145290    2566 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145294    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145297    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145300    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145303    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145306    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145309    2566 flags.go:64] FLAG: --help="false"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145313    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145316    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145319    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145322    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145325    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145328    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145331    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145334    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145336    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145339    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:50:00.150726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145342    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145345    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145348    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145351    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145354    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145356    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145359    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145362    2566 flags.go:64] FLAG: --lock-file=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145365    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145368    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145370    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145376    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145380    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145384    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145387    2566 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145390    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145393    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145396    2566 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145400    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145404    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145407    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145411    2566 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145414    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145417    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145420    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:50:00.151300 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145423    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145426    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145429    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145431    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145439    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145442    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145445    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145448    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145451    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145457    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145460    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145463    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145466    2566 flags.go:64] FLAG: --port="10250"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145469    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145472    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-052bf6eb47fe84ac3"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145475    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145478    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145481    2566 flags.go:64] FLAG: --register-node="true"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145484    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145487    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145491    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145495    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145498    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145501    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145505    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:50:00.151967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145508    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145511    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145514    2566 flags.go:64] FLAG: --runonce="false"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145517    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145520    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145523    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145526    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145529    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145532    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145535    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145538    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145540    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145544    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145547    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145549    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145552    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145555    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145558    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145563    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145566    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145569    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145573    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145576    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145578    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145581    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:50:00.152571 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145584    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145586    2566 flags.go:64] FLAG: --v="2"
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145590    2566 flags.go:64] FLAG: --version="false"
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145596    2566 flags.go:64] FLAG: --vmodule=""
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145601    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.145604    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145723    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145727    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145732    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145735    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145738    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145741    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145744    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145747    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145749    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145752    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145755    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145758    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145761    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145763    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145766    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.153207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145768    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145771    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145773    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145776    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145778    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145781    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145783    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145786    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145788    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145791    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145793    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145795    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145798    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145802    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145806    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145809    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145827    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145830    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145833    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.153723 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145836    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145839    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145841    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145844    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145846    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145848    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145851    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145854    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145856    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145858    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145861    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145863    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145866    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145869    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145871    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145874    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145877    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145879    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145882    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145884    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.154235 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145886    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145889    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145891    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145894    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145896    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145899    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145901    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145904    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145906    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145910    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145912    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145915    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145917    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145920    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145922    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145926    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145930    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145932    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145934    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145937    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.154722 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145939    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145942    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145944    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145947    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145949    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145952    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145961    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145966    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145969    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145972    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145975    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145977    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145979    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145982    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145984    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145987    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145989    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145992    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145994    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145996    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.155230 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.145999    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.146001    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.146005    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.146008    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.146016    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.146526    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.153388    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.153405    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.154061    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.154070    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154143    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154149    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154153    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154156    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.155709 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154158    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154161    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154164    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154167    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154170    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154172    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154174    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154177    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154180    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154182    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154185    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154189    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154193    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154196    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154199    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154202    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154205    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154208    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154210    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.156079 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154213    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154219    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154222    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154224    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154226    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154229    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154232    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154235    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154238    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154241    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154244    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154246    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154249    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154251    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154254    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154256    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154259    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154262    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154264    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154267    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.156527 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154270    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154272    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154275    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154278    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154280    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154282    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154285    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154287    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154290    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154292    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154295    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154298    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154300    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154302    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154306    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154309    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154311    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154314    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154317    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154319    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.157027 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154322    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154324    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154326    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154329    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154331    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154334    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154336    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154339    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154341    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154344    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154346    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154349    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154351    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154354    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154356    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154359    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154361    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154363    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154366    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154368    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.157544 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154371    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154374    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154379    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154382    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154385    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154388    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154390    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154393    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154396    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154399    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154401    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154404    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154406    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154409    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154411    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.158054 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154415    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.154421    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154537    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154541    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154544    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154547    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154550    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154553    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154555    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154558    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154561    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154563    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154566    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154569    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154571    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.158415 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154574    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154576    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154579    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154581    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154585    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154587    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154590    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154592    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154595    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154597    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154602    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154604    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154607    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154610    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154612    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154615    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154617    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154620    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154622    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.158797 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154625    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154627    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154629    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154632    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154634    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154637    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154639    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154641    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154644    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154646    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154649    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154651    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154654    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154656    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154659    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154661    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154664    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154666    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154669    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.159410 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154671    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154675    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154678    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154681    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154684    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154688    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154690    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154693    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154696    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154699    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154702    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154704    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154707    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154711    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154714    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154717    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154719    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154722    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154724    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.159900 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154727    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154729    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154732    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154735    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154737    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154740    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154742    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154745    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154747    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154749    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154752    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154755    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154758    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154760    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154763    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154767    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154769    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154772    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154774    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154777    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.160350 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154780    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154782    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154785    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154788    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154790    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154793    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154796    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154798    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:00.154800    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.154805    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.154956    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.157086    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.158527    2566 server.go:1019] "Starting client certificate rotation"
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.158630    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:00.160876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.158666    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:00.161217 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.160659    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:00.162584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.162568    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:00.170458 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.170437    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:50:00.173406 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.173387    2566 log.go:25] "Validated CRI v1 image API"
Jul 16 21:50:00.174217 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.174200    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:50:00.176690 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.176669    2566 fs.go:135] Filesystem UUIDs: map[0457a975-a85b-4070-ba80-f8157ad91a5c:/dev/nvme0n1p4 21e7f6b9-f4ec-43b1-8685-6f63dc76b490:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 16 21:50:00.176733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.176690    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:50:00.181609 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.181496    2566 manager.go:216] Machine: {Timestamp:2026-07-16 21:50:00.180336727 +0000 UTC m=+0.275848034 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097977 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec278429b5ccb7270be20f847d95780d SystemUUID:ec278429-b5cc-b727-0be2-0f847d95780d BootID:abcbf896-a0fc-436d-9edc-cb8835574306 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:c2:d5:6c:ad Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:c2:d5:6c:ad Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:d8:1b:8b:3d:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:50:00.181609 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.181603    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:50:00.181715 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.181688    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:50:00.182625 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.182598    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:50:00.182762 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.182628    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-100.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:50:00.183495 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.183483    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:50:00.183495 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.183497    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:50:00.183570 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.183505    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:00.183570 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.183520    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:00.184843 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.184828    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:50:00.186778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.186759    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:00.188599 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188588    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:00.188722 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188712    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:50:00.188763 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188741    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:50:00.188763 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188758    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:50:00.188838 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188775    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:50:00.188838 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188785    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:50:00.188838 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.188797    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:50:00.189247 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.189232    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:00.189297 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.189261    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:00.191199 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.191173    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:50:00.192935 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.192919    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:50:00.193029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.192958    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:50:00.194434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194420    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:50:00.194434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194439    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:50:00.194434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194449    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194458    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194470    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194478    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194489    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194498    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194509    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194517    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194529    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:50:00.194602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.194538    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:50:00.195103 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.195091    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:50:00.195157 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.195105    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:50:00.198503 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.198488    2566 server.go:1268] "Started kubelet"
Jul 16 21:50:00.198598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.198557    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:50:00.199416 ip-10-0-143-100 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:50:00.199546 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.199385    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:50:00.199718 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.199612    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:50:00.200210 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.200191    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-100.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:50:00.200835 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.200806    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:50:00.200905 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.200842    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:50:00.200905 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.200857    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:50:00.203661 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.203637    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:50:00.204932 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.204909    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:50:00.209805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.209787    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:50:00.209917 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.209838    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:00.210015 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.209997    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.210405 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.209604    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-100.ec2.internal.18c2e38db75e914c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-100.ec2.internal,UID:ip-10-0-143-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-100.ec2.internal,},FirstTimestamp:2026-07-16 21:50:00.198459724 +0000 UTC m=+0.293971032,LastTimestamp:2026-07-16 21:50:00.198459724 +0000 UTC m=+0.293971032,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-100.ec2.internal,}"
Jul 16 21:50:00.210578 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210467    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:50:00.210578 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210489    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:50:00.210578 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210493    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:50:00.210723 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210638    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:50:00.210723 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210649    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:50:00.210923 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210905    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:50:00.210982 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210925    2566 factory.go:55] Registering systemd factory
Jul 16 21:50:00.210982 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.210934    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:50:00.211135 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.211123    2566 factory.go:153] Registering CRI-O factory
Jul 16 21:50:00.211170 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.211149    2566 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:50:00.211170 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.211168    2566 factory.go:103] Registering Raw factory
Jul 16 21:50:00.211231 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.211178    2566 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:50:00.211686 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.211661    2566 manager.go:349] Starting recovery of all containers
Jul 16 21:50:00.215710 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.215679    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:50:00.222097 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.222063    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-100.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:50:00.222240 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.222225    2566 manager.go:354] Recovery completed
Jul 16 21:50:00.226187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.226167    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bzb6j"
Jul 16 21:50:00.227516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.227503    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.229383 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229233    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.229450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229401    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.229450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229418    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.229980 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229965    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:50:00.229980 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229979    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:50:00.230066 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229980    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.230066 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.229996    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:00.230726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.230711    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.230783 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.230738    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.230783 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.230748    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.231272 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.231193    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-100.ec2.internal.18c2e38db93673e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-100.ec2.internal,UID:ip-10-0-143-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-100.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-100.ec2.internal,},FirstTimestamp:2026-07-16 21:50:00.22938519 +0000 UTC m=+0.324896498,LastTimestamp:2026-07-16 21:50:00.22938519 +0000 UTC m=+0.324896498,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-100.ec2.internal,}"
Jul 16 21:50:00.232376 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.232361    2566 policy_none.go:49] "None policy: Start"
Jul 16 21:50:00.232436 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.232388    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:50:00.232436 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.232408    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:50:00.234489 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.234476    2566 policy_none.go:47] "Start"
Jul 16 21:50:00.236620 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.236603    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bzb6j"
Jul 16 21:50:00.243131 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.243063    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-100.ec2.internal.18c2e38db936c9bd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-100.ec2.internal,UID:ip-10-0-143-100.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-100.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-100.ec2.internal,},FirstTimestamp:2026-07-16 21:50:00.229407165 +0000 UTC m=+0.324918473,LastTimestamp:2026-07-16 21:50:00.229407165 +0000 UTC m=+0.324918473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-100.ec2.internal,}"
Jul 16 21:50:00.271661 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.271636    2566 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:50:00.271793 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.271681    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:50:00.271793 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.271691    2566 server.go:87] "Starting device plugin registration server"
Jul 16 21:50:00.272075 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.272062    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:50:00.272112 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.272079    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:50:00.272182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.272165    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:50:00.272283 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.272253    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:50:00.272283 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.272263    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:50:00.272898 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.272879    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:50:00.273002 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.272934    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.311805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.311774    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:50:00.317681 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.313231    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:50:00.317681 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.313275    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:50:00.317681 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.313305    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:50:00.317681 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.313342    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:50:00.317681 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.317067    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:00.373208 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.373161    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.374144 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.374125    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.374244 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.374155    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.374244 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.374166    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.374244 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.374190    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.387201 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.387144    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.387201 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.387174    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-100.ec2.internal\": node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.413455 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.413418    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"]
Jul 16 21:50:00.413548 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.413509    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.414078 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.414057    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.415076 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.415058    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.415156 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.415092    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.415156 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.415102    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.417476 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.417464    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.417602 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.417586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.417658 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.417624    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.418234 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418217    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.418321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418233    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.418321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418248    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.418321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418255    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.418321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418259    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.418321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.418267    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.420520 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.420504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.420588 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.420534    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.421256 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.421241    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.421340 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.421276    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.421340 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.421292    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.445707 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.445675    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-100.ec2.internal\" not found" node="ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.450217 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.450196    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-100.ec2.internal\" not found" node="ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.512002 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.511973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.512002 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.512004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.512200 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.512027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc363aecea93ceaa1e0b8c8b651a90ec-config\") pod \"kube-apiserver-proxy-ip-10-0-143-100.ec2.internal\" (UID: \"dc363aecea93ceaa1e0b8c8b651a90ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.515078 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.515056    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.612450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.612450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.612624 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc363aecea93ceaa1e0b8c8b651a90ec-config\") pod \"kube-apiserver-proxy-ip-10-0-143-100.ec2.internal\" (UID: \"dc363aecea93ceaa1e0b8c8b651a90ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.612624 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc363aecea93ceaa1e0b8c8b651a90ec-config\") pod \"kube-apiserver-proxy-ip-10-0-143-100.ec2.internal\" (UID: \"dc363aecea93ceaa1e0b8c8b651a90ec\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.612624 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.612624 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.612540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b806dca4439d4f983d86436dbc63a169-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal\" (UID: \"b806dca4439d4f983d86436dbc63a169\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.615536 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.615506    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.715724 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.715692    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.748929 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.748905    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.752731 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:00.752709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:00.816792 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.816751    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:00.917311 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:00.917259    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.017943 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:01.017850    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.119092 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:01.119057    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.158360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.158329    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:50:01.158841 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.158565    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:50:01.219293 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:01.219254    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.243294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.243244    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:45:01 +0000 UTC" deadline="2028-02-06 13:57:31.379579516 +0000 UTC"
Jul 16 21:50:01.243294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.243290    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13672h7m30.136294164s"
Jul 16 21:50:01.319733 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:01.319645    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.383705 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.383670    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:01.384744 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.384706    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:01.385797 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.385769    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:01.385927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.385813    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:01.385927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.385831    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:01.420546 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:01.420513    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-100.ec2.internal\" not found"
Jul 16 21:50:01.468113 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.468079    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:01.510611 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.510570    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:01.525671 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.525638    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:01.527436 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.527403    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal"
Jul 16 21:50:01.542100 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.542071    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:01.612207 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:01.612164    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc363aecea93ceaa1e0b8c8b651a90ec.slice/crio-d68845877d596541d62d4e1dc39c9b5dff1b6cd8105d38ee43ade0fa5680d2f8 WatchSource:0}: Error finding container d68845877d596541d62d4e1dc39c9b5dff1b6cd8105d38ee43ade0fa5680d2f8: Status 404 returned error can't find the container with id d68845877d596541d62d4e1dc39c9b5dff1b6cd8105d38ee43ade0fa5680d2f8
Jul 16 21:50:01.612615 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:01.612591    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb806dca4439d4f983d86436dbc63a169.slice/crio-2a7fced0cdd7bdc0453341f4115005e9bfa77c453351a735ace084bda112478b WatchSource:0}: Error finding container 2a7fced0cdd7bdc0453341f4115005e9bfa77c453351a735ace084bda112478b: Status 404 returned error can't find the container with id 2a7fced0cdd7bdc0453341f4115005e9bfa77c453351a735ace084bda112478b
Jul 16 21:50:01.617687 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:01.617663    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:50:02.167080 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.167043    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:02.192049 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.192018    2566 apiserver.go:52] "Watching apiserver"
Jul 16 21:50:02.198797 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.198773    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:50:02.201103 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.201078    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-zm8gq","openshift-network-operator/iptables-alerter-xb5jg","kube-system/global-pull-secret-syncer-wjrxm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal","openshift-multus/multus-additional-cni-plugins-c899z","openshift-multus/network-metrics-daemon-jsh7v","openshift-network-diagnostics/network-check-target-8twsk","openshift-ovn-kubernetes/ovnkube-node-jndh2","kube-system/konnectivity-agent-kkkpd","kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rr7zg","openshift-dns/node-resolver-9mc6r","openshift-image-registry/node-ca-fs66g"]
Jul 16 21:50:02.205203 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.205179    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.205300 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.205272    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:02.207426 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.207404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:02.207499 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.207472    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:02.209673 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.209655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.209752 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.209702    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:02.211066 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.211047    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:02.213924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.213897    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.216308 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.216283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.216671 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.216641    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4gpjd\""
Jul 16 21:50:02.216974 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.216954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:50:02.217082 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.217065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.217129 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.217094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.218611 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.218593    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.219581 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.219563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.219936 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.219869    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.219936 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.219902    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:50:02.219936 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.219919    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jmtt8\""
Jul 16 21:50:02.220143 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.220123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:50:02.221250 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.221360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221232    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:50:02.221459 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bdhdd\""
Jul 16 21:50:02.221717 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221696    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:50:02.221884 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.221945 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.221945 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lsk9\" (UniqueName: \"kubernetes.io/projected/fdc51d15-a2f4-40e6-b973-11dd4094fe35-kube-api-access-2lsk9\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.222029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-dbus\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.222029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-socket-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.221990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-registration-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222029 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-device-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:02.222251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-kubelet-config\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.222251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-sys-fs\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q75g\" (UniqueName: \"kubernetes.io/projected/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kube-api-access-7q75g\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.222251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.222171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.223824 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.223800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.224438 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.224420    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:50:02.225023 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.224998    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:50:02.225111 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.225094    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:02.225172 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.225155    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:50:02.225580 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.225561    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.225654 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.225600    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.225708 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.225674    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mk8fp\""
Jul 16 21:50:02.226284 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.226263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:50:02.226362 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.226299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.227022 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.227007    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.227927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.227899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-csbvl\""
Jul 16 21:50:02.228486 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.228468    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.228999 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.228976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.229264 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.229246    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.229312 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.229263    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhhcn\""
Jul 16 21:50:02.229703 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.229687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.229803 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.229788    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:50:02.232107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.232091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.233270 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.233252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:50:02.234168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.234150    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gpdss\""
Jul 16 21:50:02.234253 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.234176    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:50:02.234689 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.234670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.234784 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.234709    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.234784 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.234714    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:50:02.235213 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.235192    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:02.235310 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.235253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.235638 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.235622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7xjp4\""
Jul 16 21:50:02.239620 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.239600    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.239719 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.239626    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-42thw\""
Jul 16 21:50:02.240511 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.240493    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.270373 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.270339    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dtljz"
Jul 16 21:50:02.281141 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.281116    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dtljz"
Jul 16 21:50:02.311981 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.311943    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:50:02.320256 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.320191    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal" event={"ID":"dc363aecea93ceaa1e0b8c8b651a90ec","Type":"ContainerStarted","Data":"d68845877d596541d62d4e1dc39c9b5dff1b6cd8105d38ee43ade0fa5680d2f8"}
Jul 16 21:50:02.322360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-os-release\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.322360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal" event={"ID":"b806dca4439d4f983d86436dbc63a169","Type":"ContainerStarted","Data":"2a7fced0cdd7bdc0453341f4115005e9bfa77c453351a735ace084bda112478b"}
Jul 16 21:50:02.322578 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-bin\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.322641 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-modprobe-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.322699 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-system-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.322766 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-k8s-cni-cncf-io\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.322766 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ea52686-3288-4855-aef7-2ba5ac429dbf-hosts-file\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.322891 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-conf\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.322950 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-registration-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.323052 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.322926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz7gs\" (UniqueName: \"kubernetes.io/projected/e1304cb1-326c-4c14-8464-201068718829-kube-api-access-rz7gs\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.323118 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323025    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-registration-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.323118 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-var-lib-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323222 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-node-log\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323222 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-netd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-iptables-alerter-script\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.323321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323259    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-host-slash\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.323321 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-kubelet-config\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.323422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323361    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-kubelet-config\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.323422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-slash\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-script-lib\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-tuned\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.323500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-socket-dir-parent\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.323644 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-hostroot\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.323644 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323540    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-etc-kubernetes\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.323644 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-ovn\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323644 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-env-overrides\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.323644 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-lib-modules\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.323805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-host\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.323805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-netns\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.323805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-multus\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.323805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.323909    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-etc-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-log-socket\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.323985    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.823961688 +0000 UTC m=+2.919473001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.323985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.324019 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-sys\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea52686-3288-4855-aef7-2ba5ac429dbf-tmp-dir\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-os-release\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa0b617-4199-4471-b583-fd0d97634b58-ovn-node-metrics-cert\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-kube-api-access-bxbx8\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtbv\" (UniqueName: \"kubernetes.io/projected/924fa652-0d0a-4d38-ae73-d20387be6f26-kube-api-access-lxtbv\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-sys-fs\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-system-cni-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-cnibin\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-sys-fs\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.324355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-config\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkgpf\" (UniqueName: \"kubernetes.io/projected/ffa0b617-4199-4471-b583-fd0d97634b58-kube-api-access-dkgpf\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-cni-binary-copy\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1304cb1-326c-4c14-8464-201068718829-serviceca\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-var-lib-kubelet\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-daemon-config\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-multus-certs\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324617    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-etc-selinux\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-bin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/936c198b-a37a-4426-bc7d-9cbcd336282d-konnectivity-ca\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.325030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xbz\" (UniqueName: \"kubernetes.io/projected/10098757-a07b-47c0-9786-61b5704881b0-kube-api-access-v7xbz\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-dbus\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-socket-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-device-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-systemd-units\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-netns\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324814    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-conf-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q75g\" (UniqueName: \"kubernetes.io/projected/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kube-api-access-7q75g\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-systemd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-dbus\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78drx\" (UniqueName: \"kubernetes.io/projected/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-kube-api-access-78drx\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.324979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/936c198b-a37a-4426-bc7d-9cbcd336282d-agent-certs\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-kubelet\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-kubernetes\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325052    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-systemd\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.325682 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-run\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-socket-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-tmp\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-device-dir\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lsk9\" (UniqueName: \"kubernetes.io/projected/fdc51d15-a2f4-40e6-b973-11dd4094fe35-kube-api-access-2lsk9\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-kubelet\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttzj\" (UniqueName: \"kubernetes.io/projected/6ea52686-3288-4855-aef7-2ba5ac429dbf-kube-api-access-8ttzj\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1304cb1-326c-4c14-8464-201068718829-host\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysconfig\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.325329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-cnibin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.325500    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.326422 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.325551    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.825533211 +0000 UTC m=+2.921044513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.343429 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.343397    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:02.343429 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.343433    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:02.343665 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.343449    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:02.343665 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.343526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:02.84350517 +0000 UTC m=+2.939016468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:02.343772 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.343680    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:50:02.347973 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.347939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lsk9\" (UniqueName: \"kubernetes.io/projected/fdc51d15-a2f4-40e6-b973-11dd4094fe35-kube-api-access-2lsk9\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.348105 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.347992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q75g\" (UniqueName: \"kubernetes.io/projected/17cd8f8a-cba8-4ad1-948d-ea6dece60d85-kube-api-access-7q75g\") pod \"aws-ebs-csi-driver-node-r4bfh\" (UID: \"17cd8f8a-cba8-4ad1-948d-ea6dece60d85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.426353 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-kubelet\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426353 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttzj\" (UniqueName: \"kubernetes.io/projected/6ea52686-3288-4855-aef7-2ba5ac429dbf-kube-api-access-8ttzj\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-kubelet\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1304cb1-326c-4c14-8464-201068718829-host\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysconfig\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-cnibin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-os-release\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-bin\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426607    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-modprobe-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-system-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysconfig\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-k8s-cni-cncf-io\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1304cb1-326c-4c14-8464-201068718829-host\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ea52686-3288-4855-aef7-2ba5ac429dbf-hosts-file\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-bin\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-conf\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-cnibin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz7gs\" (UniqueName: \"kubernetes.io/projected/e1304cb1-326c-4c14-8464-201068718829-kube-api-access-rz7gs\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-var-lib-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-os-release\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-node-log\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ea52686-3288-4855-aef7-2ba5ac429dbf-hosts-file\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-netd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426892    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-iptables-alerter-script\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-host-slash\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.426928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-modprobe-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-slash\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-system-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-script-lib\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.426995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-k8s-cni-cncf-io\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-tuned\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-cni-netd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-socket-dir-parent\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-hostroot\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-etc-kubernetes\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-ovn\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427123    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-conf\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-env-overrides\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-var-lib-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-node-log\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-lib-modules\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-host\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-netns\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.427863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-multus\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-etc-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-log-socket\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-sys\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea52686-3288-4855-aef7-2ba5ac429dbf-tmp-dir\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-os-release\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa0b617-4199-4471-b583-fd0d97634b58-ovn-node-metrics-cert\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-kube-api-access-bxbx8\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtbv\" (UniqueName: \"kubernetes.io/projected/924fa652-0d0a-4d38-ae73-d20387be6f26-kube-api-access-lxtbv\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-system-cni-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-env-overrides\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-cnibin\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-config\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.428733 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkgpf\" (UniqueName: \"kubernetes.io/projected/ffa0b617-4199-4471-b583-fd0d97634b58-kube-api-access-dkgpf\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.427984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-cni-binary-copy\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-host-slash\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1304cb1-326c-4c14-8464-201068718829-serviceca\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-var-lib-kubelet\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-daemon-config\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-multus-certs\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-bin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/936c198b-a37a-4426-bc7d-9cbcd336282d-konnectivity-ca\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xbz\" (UniqueName: \"kubernetes.io/projected/10098757-a07b-47c0-9786-61b5704881b0-kube-api-access-v7xbz\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-systemd-units\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-netns\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-conf-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e1304cb1-326c-4c14-8464-201068718829-serviceca\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.429584 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-systemd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78drx\" (UniqueName: \"kubernetes.io/projected/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-kube-api-access-78drx\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/936c198b-a37a-4426-bc7d-9cbcd336282d-agent-certs\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-kubelet\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-lib-modules\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-kubernetes\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-host\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-systemd\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-netns\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-run\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-multus\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-tmp\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-log-socket\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-etc-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-sys\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.428807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-slash\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea52686-3288-4855-aef7-2ba5ac429dbf-tmp-dir\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.430483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-os-release\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-hostroot\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-socket-dir-parent\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-system-cni-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-binary-copy\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429768    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-cnibin\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.429863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-script-lib\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-openvswitch\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-var-lib-kubelet\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-var-lib-cni-bin\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10098757-a07b-47c0-9786-61b5704881b0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-ovn-kubernetes\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-cni-binary-copy\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-etc-kubernetes\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-tuned\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.431387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-cni-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-systemd-units\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430719    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/936c198b-a37a-4426-bc7d-9cbcd336282d-konnectivity-ca\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-kubelet\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-systemd\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-kubernetes\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-iptables-alerter-script\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-conf-dir\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-ovn\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-etc-sysctl-d\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-run\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/924fa652-0d0a-4d38-ae73-d20387be6f26-host-run-multus-certs\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.430907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-run-systemd\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.431051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffa0b617-4199-4471-b583-fd0d97634b58-host-run-netns\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.431429    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffa0b617-4199-4471-b583-fd0d97634b58-ovnkube-config\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.431874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10098757-a07b-47c0-9786-61b5704881b0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.432235 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.431922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/924fa652-0d0a-4d38-ae73-d20387be6f26-multus-daemon-config\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.433348 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.433300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-tmp\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.434938 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.434917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffa0b617-4199-4471-b583-fd0d97634b58-ovn-node-metrics-cert\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.435354 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.435313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/936c198b-a37a-4426-bc7d-9cbcd336282d-agent-certs\") pod \"konnectivity-agent-kkkpd\" (UID: \"936c198b-a37a-4426-bc7d-9cbcd336282d\") " pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.448863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.448831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xbz\" (UniqueName: \"kubernetes.io/projected/10098757-a07b-47c0-9786-61b5704881b0-kube-api-access-v7xbz\") pod \"multus-additional-cni-plugins-c899z\" (UID: \"10098757-a07b-47c0-9786-61b5704881b0\") " pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.449830 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.449799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbx8\" (UniqueName: \"kubernetes.io/projected/374fb0cc-21cb-44be-b5bf-d84d5a9ac13f-kube-api-access-bxbx8\") pod \"iptables-alerter-xb5jg\" (UID: \"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f\") " pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.452089 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.452063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78drx\" (UniqueName: \"kubernetes.io/projected/4c6e0e81-75dd-4ffa-b754-a25ae81d565c-kube-api-access-78drx\") pod \"tuned-rr7zg\" (UID: \"4c6e0e81-75dd-4ffa-b754-a25ae81d565c\") " pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.452792 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.452746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtbv\" (UniqueName: \"kubernetes.io/projected/924fa652-0d0a-4d38-ae73-d20387be6f26-kube-api-access-lxtbv\") pod \"multus-zm8gq\" (UID: \"924fa652-0d0a-4d38-ae73-d20387be6f26\") " pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.453677 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.453628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkgpf\" (UniqueName: \"kubernetes.io/projected/ffa0b617-4199-4471-b583-fd0d97634b58-kube-api-access-dkgpf\") pod \"ovnkube-node-jndh2\" (UID: \"ffa0b617-4199-4471-b583-fd0d97634b58\") " pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.455739 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.455707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz7gs\" (UniqueName: \"kubernetes.io/projected/e1304cb1-326c-4c14-8464-201068718829-kube-api-access-rz7gs\") pod \"node-ca-fs66g\" (UID: \"e1304cb1-326c-4c14-8464-201068718829\") " pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.456461 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.456440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttzj\" (UniqueName: \"kubernetes.io/projected/6ea52686-3288-4855-aef7-2ba5ac429dbf-kube-api-access-8ttzj\") pod \"node-resolver-9mc6r\" (UID: \"6ea52686-3288-4855-aef7-2ba5ac429dbf\") " pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.531596 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.529791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh"
Jul 16 21:50:02.532496 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.532460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zm8gq"
Jul 16 21:50:02.540937 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.540913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c899z"
Jul 16 21:50:02.552308 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.547148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:02.552308 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.550817    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cd8f8a_cba8_4ad1_948d_ea6dece60d85.slice/crio-a46fc47017b0eb62d1ef7d5f849eea737adce9a17638009b4ff082d238098137 WatchSource:0}: Error finding container a46fc47017b0eb62d1ef7d5f849eea737adce9a17638009b4ff082d238098137: Status 404 returned error can't find the container with id a46fc47017b0eb62d1ef7d5f849eea737adce9a17638009b4ff082d238098137
Jul 16 21:50:02.552572 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.552496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924fa652_0d0a_4d38_ae73_d20387be6f26.slice/crio-79f0b05d7d434910cdb7e05b2e8030ada938cd35f780761204405976c47c60a7 WatchSource:0}: Error finding container 79f0b05d7d434910cdb7e05b2e8030ada938cd35f780761204405976c47c60a7: Status 404 returned error can't find the container with id 79f0b05d7d434910cdb7e05b2e8030ada938cd35f780761204405976c47c60a7
Jul 16 21:50:02.557732 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.557707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9mc6r"
Jul 16 21:50:02.562575 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.562540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xb5jg"
Jul 16 21:50:02.562884 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.562854    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10098757_a07b_47c0_9786_61b5704881b0.slice/crio-1adfe714617d9a53c1b534d06d3f99fd7ed2aca010b8b141acd11eee9de2bb7f WatchSource:0}: Error finding container 1adfe714617d9a53c1b534d06d3f99fd7ed2aca010b8b141acd11eee9de2bb7f: Status 404 returned error can't find the container with id 1adfe714617d9a53c1b534d06d3f99fd7ed2aca010b8b141acd11eee9de2bb7f
Jul 16 21:50:02.563917 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.563875    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa0b617_4199_4471_b583_fd0d97634b58.slice/crio-7286e641dad73b4302e9244295ad4ae386f29537055c06120f881cd761551bff WatchSource:0}: Error finding container 7286e641dad73b4302e9244295ad4ae386f29537055c06120f881cd761551bff: Status 404 returned error can't find the container with id 7286e641dad73b4302e9244295ad4ae386f29537055c06120f881cd761551bff
Jul 16 21:50:02.574036 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.574005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:02.575914 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.575871    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea52686_3288_4855_aef7_2ba5ac429dbf.slice/crio-44fabd087b58497e4b098a23b4f4c3be75ad2828ecb3b0917eb85eab6732ce88 WatchSource:0}: Error finding container 44fabd087b58497e4b098a23b4f4c3be75ad2828ecb3b0917eb85eab6732ce88: Status 404 returned error can't find the container with id 44fabd087b58497e4b098a23b4f4c3be75ad2828ecb3b0917eb85eab6732ce88
Jul 16 21:50:02.587481 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.587444    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fs66g"
Jul 16 21:50:02.589280 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.589250    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c198b_a37a_4426_bc7d_9cbcd336282d.slice/crio-c8c2aadc539263f763b241bd292fffef4754bf515f757d5a93be94b685278827 WatchSource:0}: Error finding container c8c2aadc539263f763b241bd292fffef4754bf515f757d5a93be94b685278827: Status 404 returned error can't find the container with id c8c2aadc539263f763b241bd292fffef4754bf515f757d5a93be94b685278827
Jul 16 21:50:02.593943 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.593921    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rr7zg"
Jul 16 21:50:02.601869 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.601838    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1304cb1_326c_4c14_8464_201068718829.slice/crio-fffbb8dd69b5c7b53b967ebf1418918f7ea93aff011dfd43d4881e464861131b WatchSource:0}: Error finding container fffbb8dd69b5c7b53b967ebf1418918f7ea93aff011dfd43d4881e464861131b: Status 404 returned error can't find the container with id fffbb8dd69b5c7b53b967ebf1418918f7ea93aff011dfd43d4881e464861131b
Jul 16 21:50:02.610555 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:02.610526    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6e0e81_75dd_4ffa_b754_a25ae81d565c.slice/crio-ac063b221908061677cc5c2d112db9f39d1b69d73fcf7f70d2c54467c20afa2b WatchSource:0}: Error finding container ac063b221908061677cc5c2d112db9f39d1b69d73fcf7f70d2c54467c20afa2b: Status 404 returned error can't find the container with id ac063b221908061677cc5c2d112db9f39d1b69d73fcf7f70d2c54467c20afa2b
Jul 16 21:50:02.833177 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.833139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:02.833369 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.833220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:02.833369 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.833286    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.833369 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.833336    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.833541 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.833377    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.83335452 +0000 UTC m=+3.928865834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.833541 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.833401    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.833389121 +0000 UTC m=+3.928900423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.934362 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:02.934307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:02.934543 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.934499    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:02.934543 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.934524    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:02.934543 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.934534    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:02.934649 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:02.934597    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.934580973 +0000 UTC m=+4.030092269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:03.286703 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.286652    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:02 +0000 UTC" deadline="2028-01-29 21:16:12.813895324 +0000 UTC"
Jul 16 21:50:03.286703 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.286693    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13487h26m9.527205604s"
Jul 16 21:50:03.314197 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.314162    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:03.314363 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.314321    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:03.325391 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.325349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rr7zg" event={"ID":"4c6e0e81-75dd-4ffa-b754-a25ae81d565c","Type":"ContainerStarted","Data":"ac063b221908061677cc5c2d112db9f39d1b69d73fcf7f70d2c54467c20afa2b"}
Jul 16 21:50:03.326555 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.326526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kkkpd" event={"ID":"936c198b-a37a-4426-bc7d-9cbcd336282d","Type":"ContainerStarted","Data":"c8c2aadc539263f763b241bd292fffef4754bf515f757d5a93be94b685278827"}
Jul 16 21:50:03.327730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.327705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xb5jg" event={"ID":"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f","Type":"ContainerStarted","Data":"8764b24a0367182c5f6c56fa182f8fd09dc37f7ea4a4316bb98d23e669932692"}
Jul 16 21:50:03.328959 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.328933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"7286e641dad73b4302e9244295ad4ae386f29537055c06120f881cd761551bff"}
Jul 16 21:50:03.330175 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.330141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerStarted","Data":"1adfe714617d9a53c1b534d06d3f99fd7ed2aca010b8b141acd11eee9de2bb7f"}
Jul 16 21:50:03.331646 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.331620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zm8gq" event={"ID":"924fa652-0d0a-4d38-ae73-d20387be6f26","Type":"ContainerStarted","Data":"79f0b05d7d434910cdb7e05b2e8030ada938cd35f780761204405976c47c60a7"}
Jul 16 21:50:03.332857 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.332817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh" event={"ID":"17cd8f8a-cba8-4ad1-948d-ea6dece60d85","Type":"ContainerStarted","Data":"a46fc47017b0eb62d1ef7d5f849eea737adce9a17638009b4ff082d238098137"}
Jul 16 21:50:03.334009 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.333977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fs66g" event={"ID":"e1304cb1-326c-4c14-8464-201068718829","Type":"ContainerStarted","Data":"fffbb8dd69b5c7b53b967ebf1418918f7ea93aff011dfd43d4881e464861131b"}
Jul 16 21:50:03.335207 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.335177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9mc6r" event={"ID":"6ea52686-3288-4855-aef7-2ba5ac429dbf","Type":"ContainerStarted","Data":"44fabd087b58497e4b098a23b4f4c3be75ad2828ecb3b0917eb85eab6732ce88"}
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.841909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.841995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.842119    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.842197    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:05.842172337 +0000 UTC m=+5.937683655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.842276    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:03.842646 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.842307    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:05.842298929 +0000 UTC m=+5.937810225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:03.943220 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:03.943164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:03.943425 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.943401    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:03.943534 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.943430    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:03.943534 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.943457    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:03.943534 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:03.943523    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:05.943502846 +0000 UTC m=+6.039014147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:04.287720 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:04.287521    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:02 +0000 UTC" deadline="2028-01-16 04:50:59.57520761 +0000 UTC"
Jul 16 21:50:04.287720 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:04.287699    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13159h0m55.287513615s"
Jul 16 21:50:04.314087 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:04.314048    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:04.314274 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:04.314093    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:04.314274 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:04.314183    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:04.314274 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:04.314250    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:04.359183 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:04.358300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal" event={"ID":"dc363aecea93ceaa1e0b8c8b651a90ec","Type":"ContainerStarted","Data":"bb76fcf5bc54e7b21be397df70f5a92f19893807f452064a6a13135879470cdd"}
Jul 16 21:50:05.314579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.314539    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:05.315114 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.314693    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:05.375934 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.375786    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal" event={"ID":"b806dca4439d4f983d86436dbc63a169","Type":"ContainerStarted","Data":"0140f3a72d122fb701826df6686181992f4087319f436912d09e374985067b5b"}
Jul 16 21:50:05.394510 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.393464    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-100.ec2.internal" podStartSLOduration=3.393439404 podStartE2EDuration="3.393439404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:04.381305812 +0000 UTC m=+4.476817139" watchObservedRunningTime="2026-07-16 21:50:05.393439404 +0000 UTC m=+5.488950727"
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.861325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.861399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.861539    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.861605    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:09.861583973 +0000 UTC m=+9.957095277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.862066    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:05.862159 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.862122    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:09.862105569 +0000 UTC m=+9.957616871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:05.963390 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:05.962663    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:05.963390 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.962847    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:05.963390 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.962885    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:05.963390 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.962900    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:05.963390 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:05.962966    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:09.962945749 +0000 UTC m=+10.058457049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:06.317500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:06.316691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:06.317500 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:06.316829    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:06.317500 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:06.317299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:06.317500 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:06.317407    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:06.391118 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:06.391077    2566 generic.go:356] "Generic (PLEG): container finished" podID="b806dca4439d4f983d86436dbc63a169" containerID="0140f3a72d122fb701826df6686181992f4087319f436912d09e374985067b5b" exitCode=0
Jul 16 21:50:06.391296 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:06.391143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal" event={"ID":"b806dca4439d4f983d86436dbc63a169","Type":"ContainerDied","Data":"0140f3a72d122fb701826df6686181992f4087319f436912d09e374985067b5b"}
Jul 16 21:50:07.314285 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:07.314250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:07.314490 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:07.314408    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:08.316160 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:08.316120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:08.316678 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:08.316286    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:08.316754 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:08.316724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:08.316859 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:08.316830    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:09.313603 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:09.313564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:09.313808 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:09.313731    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:09.900253 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:09.900206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:09.900745 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:09.900302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:09.900745 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:09.900390    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:09.900745 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:09.900403    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:09.900745 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:09.900470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:17.900448475 +0000 UTC m=+17.995959771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:09.900745 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:09.900491    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:17.900480915 +0000 UTC m=+17.995992212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:10.001020 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:10.000966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:10.001220 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.001198    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:10.001286 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.001228    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:10.001286 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.001243    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:10.001400 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.001308    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.001286047 +0000 UTC m=+18.096797362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:10.314694 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:10.314597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:10.314855 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.314715    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:10.314855 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:10.314730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:10.315023 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:10.314886    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:11.313702 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:11.313661    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:11.314174 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:11.313798    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:12.314951 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:12.314867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:12.314951 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:12.314896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:12.315418 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:12.315009    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:12.315418 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:12.315130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:13.314542 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:13.314508    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:13.314737 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:13.314628    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:14.313649 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:14.313628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:14.314027 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:14.313629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:14.314027 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:14.313767    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:14.314027 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:14.313807    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:15.314357 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:15.314309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:15.314802 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:15.314447    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:16.313979 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:16.313939    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:16.314142 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:16.314065    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:16.314142 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:16.314130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:16.314260 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:16.314237    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:17.314049 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:17.314019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:17.314476 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:17.314154    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:17.964400 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:17.964365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:17.964580 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:17.964426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:17.964580 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:17.964543    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:17.964698 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:17.964604    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.964584896 +0000 UTC m=+34.060096193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:17.964698 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:17.964665    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:17.964804 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:17.964701    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:33.964690563 +0000 UTC m=+34.060201861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:18.065346 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:18.065293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:18.065537 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.065511    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:18.065591 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.065543    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:18.065591 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.065554    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.065691 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.065607    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:34.065593642 +0000 UTC m=+34.161104936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.314095 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:18.314023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:18.314478 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.314143    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:18.314478 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:18.314234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:18.314478 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:18.314358    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:19.314470 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:19.314431    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:19.314943 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:19.314578    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:20.317234 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:20.317203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:20.317703 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:20.317203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:20.317703 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:20.317306    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:20.317703 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:20.317400    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:21.314449 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:21.314407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:21.314715 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:21.314564    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:22.314467 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:22.314421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:22.314940 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:22.314421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:22.314940 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:22.314541    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:22.314940 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:22.314599    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:23.314281 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.313949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:23.314392 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:23.314375    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:23.427520 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.427479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rr7zg" event={"ID":"4c6e0e81-75dd-4ffa-b754-a25ae81d565c","Type":"ContainerStarted","Data":"65ce4230750a134025570516031379fc49fa3a054b5286eda6fe602b4e698dfc"}
Jul 16 21:50:23.429993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.429957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kkkpd" event={"ID":"936c198b-a37a-4426-bc7d-9cbcd336282d","Type":"ContainerStarted","Data":"09e42da4dba550d80170c9ec1779df131a38f536884e4963d932e16d2aa32899"}
Jul 16 21:50:23.432604 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.432583    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:50:23.432974 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.432949    2566 generic.go:356] "Generic (PLEG): container finished" podID="ffa0b617-4199-4471-b583-fd0d97634b58" containerID="c66ccfd20b39022095c9a183b77501e7492ca0f76cc8d70b2890beb28fd50a8d" exitCode=1
Jul 16 21:50:23.433088 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.433010    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"efa4df9b8a32f5d521573fb30188b6952cf96a93846456ec003689ac3e946f57"}
Jul 16 21:50:23.433141 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.433089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"2154f36c22fb7de85c3a789d905f582c1c241c2498c0918b67937bb7869718e9"}
Jul 16 21:50:23.433188 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.433137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerDied","Data":"c66ccfd20b39022095c9a183b77501e7492ca0f76cc8d70b2890beb28fd50a8d"}
Jul 16 21:50:23.433228 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.433188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"1f15c156eef52661c3136ac90e77073cd15bf86b37a5a2c7839aa03545d4b862"}
Jul 16 21:50:23.434755 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.434731    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="88b48d63925fda13d426d57b9c116ce0fa39ab43902f0cde27b41f7a65493f2a" exitCode=0
Jul 16 21:50:23.434886 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.434800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"88b48d63925fda13d426d57b9c116ce0fa39ab43902f0cde27b41f7a65493f2a"}
Jul 16 21:50:23.436492 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.436341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zm8gq" event={"ID":"924fa652-0d0a-4d38-ae73-d20387be6f26","Type":"ContainerStarted","Data":"b6b12abfe581308e49df8fd1be3d171ea97d3d9e086f8eb3043faef81d3112ea"}
Jul 16 21:50:23.438805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.438782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh" event={"ID":"17cd8f8a-cba8-4ad1-948d-ea6dece60d85","Type":"ContainerStarted","Data":"93a8037204038ff8d60cc24e96f965884100c8bee1990a40b327104ea75ec985"}
Jul 16 21:50:23.440690 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.440671    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal" event={"ID":"b806dca4439d4f983d86436dbc63a169","Type":"ContainerStarted","Data":"c82aeb39e0021ca8a00dc669d418f5cf541400fe2a08236259fc3e483fc742a5"}
Jul 16 21:50:23.442279 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.442243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fs66g" event={"ID":"e1304cb1-326c-4c14-8464-201068718829","Type":"ContainerStarted","Data":"639915e036512b92cc6a053329d7d8b2841500624af9d781d252f7706643d444"}
Jul 16 21:50:23.444037 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.444015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9mc6r" event={"ID":"6ea52686-3288-4855-aef7-2ba5ac429dbf","Type":"ContainerStarted","Data":"d508265c2de953267c60c270a1e2cda5c3480571b869f194422e321c5541aabb"}
Jul 16 21:50:23.447588 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.447549    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rr7zg" podStartSLOduration=2.337832235 podStartE2EDuration="22.447537525s" totalImagesPullingTime="20.10970529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.446994523 +0000 UTC m=+23.542505840" watchObservedRunningTime="2026-07-16 21:50:23.447537525 +0000 UTC m=+23.543048874"
Jul 16 21:50:23.464972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.464949    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:23.465509 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.465484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:23.473434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.470899    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zm8gq" podStartSLOduration=2.290794343 podStartE2EDuration="22.470860038s" totalImagesPullingTime="20.180065695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.4706504 +0000 UTC m=+23.566161719" watchObservedRunningTime="2026-07-16 21:50:23.470860038 +0000 UTC m=+23.566371359"
Jul 16 21:50:23.541732 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.541667    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9mc6r" podStartSLOduration=2.42408282 podStartE2EDuration="22.541646428s" totalImagesPullingTime="20.117563608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.541121742 +0000 UTC m=+23.636633062" watchObservedRunningTime="2026-07-16 21:50:23.541646428 +0000 UTC m=+23.637157747"
Jul 16 21:50:23.542152 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.542122    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-100.ec2.internal" podStartSLOduration=21.54211188 podStartE2EDuration="21.54211188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.519208813 +0000 UTC m=+23.614720130" watchObservedRunningTime="2026-07-16 21:50:23.54211188 +0000 UTC m=+23.637623198"
Jul 16 21:50:23.567301 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.567244    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kkkpd" podStartSLOduration=2.898608518 podStartE2EDuration="22.56721896s" totalImagesPullingTime="19.668610442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.566834955 +0000 UTC m=+23.662346265" watchObservedRunningTime="2026-07-16 21:50:23.56721896 +0000 UTC m=+23.662730280"
Jul 16 21:50:23.593674 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:23.593621    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fs66g" podStartSLOduration=2.530160816 podStartE2EDuration="22.593603257s" totalImagesPullingTime="20.063442441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.593318122 +0000 UTC m=+23.688829439" watchObservedRunningTime="2026-07-16 21:50:23.593603257 +0000 UTC m=+23.689114574"
Jul 16 21:50:24.098863 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.098838    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:50:24.284897 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.284733    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:24.098859546Z","UUID":"cf5cbc52-c658-42e3-bd27-c0251f81b126","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:50:24.286416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.286395    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:50:24.286587 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.286439    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:50:24.314579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.314546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:24.314766 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:24.314680    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:24.314766 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.314750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:24.314913 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:24.314860    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:24.448894 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.448856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xb5jg" event={"ID":"374fb0cc-21cb-44be-b5bf-d84d5a9ac13f","Type":"ContainerStarted","Data":"d2ac542918df4ba2052718e024a50d4d6fe03267afbae32300294b1c67697424"}
Jul 16 21:50:24.452443 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.452420    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:50:24.452898 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.452862    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"31686a08dbd80a7e932246ee4a6692e4468e7b69ea1dbc8a82db18c3a6f3903e"}
Jul 16 21:50:24.452985 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.452958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"8c1b03927c5762795b31bdec1b83ba3af473be15f2767ee0b765505d0737f5f7"}
Jul 16 21:50:24.455224 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.455192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh" event={"ID":"17cd8f8a-cba8-4ad1-948d-ea6dece60d85","Type":"ContainerStarted","Data":"4ba39a43a7a3fba8de0966d2fc8627b9e4fef1d2a51df239bb2f7d29060c9039"}
Jul 16 21:50:24.455760 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.455736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:24.456412 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.456393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kkkpd"
Jul 16 21:50:24.488101 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:24.488050    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xb5jg" podStartSLOduration=3.3425822370000002 podStartE2EDuration="23.48803346s" totalImagesPullingTime="20.145451223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:24.468509968 +0000 UTC m=+24.564021284" watchObservedRunningTime="2026-07-16 21:50:24.48803346 +0000 UTC m=+24.583544758"
Jul 16 21:50:25.314601 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:25.314561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:25.314770 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:25.314714    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:25.459783 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:25.459733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh" event={"ID":"17cd8f8a-cba8-4ad1-948d-ea6dece60d85","Type":"ContainerStarted","Data":"eafe5fd5b6740aafd5f45e080acf505a0e08956c784f3f930f6e9a684df50934"}
Jul 16 21:50:26.314361 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:26.314317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:26.314565 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:26.314317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:26.314565 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:26.314464    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:26.314565 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:26.314551    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:26.466471 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:26.466440    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:50:26.467002 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:26.466967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"5f81f0d51ac9800c4874d824f81c71b563be566f273911c8eefce0398b344392"}
Jul 16 21:50:27.314137 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:27.313949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:27.314347 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:27.314255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:28.314152 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.313961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:28.314956 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.314029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:28.314956 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:28.314263    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:28.314956 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:28.314333    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:28.474928 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.474902    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:50:28.475247 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.475224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"03ed502f4dc9e268dff909769d60db416fe04c2d26970f871cea7de31422ddff"}
Jul 16 21:50:28.475554 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.475533    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:28.475797 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.475781    2566 scope.go:117] "RemoveContainer" containerID="c66ccfd20b39022095c9a183b77501e7492ca0f76cc8d70b2890beb28fd50a8d"
Jul 16 21:50:28.477245 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.477223    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="de97e2fe8e9d2da154aab2778b83769872394aa31ec1d04f141d6d8e6d696c9c" exitCode=0
Jul 16 21:50:28.477366 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.477269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"de97e2fe8e9d2da154aab2778b83769872394aa31ec1d04f141d6d8e6d696c9c"}
Jul 16 21:50:28.492285 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.492266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:28.507103 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:28.507046    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r4bfh" podStartSLOduration=5.696673513 podStartE2EDuration="27.507032257s" totalImagesPullingTime="21.810358744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.522398967 +0000 UTC m=+25.617910295" watchObservedRunningTime="2026-07-16 21:50:28.507032257 +0000 UTC m=+28.602543573"
Jul 16 21:50:29.316167 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.314992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:29.316167 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:29.315180    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:29.485723 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.485696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:50:29.486106 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.486081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" event={"ID":"ffa0b617-4199-4471-b583-fd0d97634b58","Type":"ContainerStarted","Data":"67de21930743d315583652fcf1f1530184923347de293ad880a0a17b1f484890"}
Jul 16 21:50:29.486288 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.486254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:29.486427 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.486319    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:29.488259 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.488232    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="5c22d8955cef1e60af6c6167c311b24f0968b85fceafca85f6b126feb8b144b4" exitCode=0
Jul 16 21:50:29.488339 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.488288    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"5c22d8955cef1e60af6c6167c311b24f0968b85fceafca85f6b126feb8b144b4"}
Jul 16 21:50:29.503248 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.503226    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:50:29.518624 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:29.518579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2" podStartSLOduration=8.299470611 podStartE2EDuration="28.518565609s" totalImagesPullingTime="20.219094998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:29.518312475 +0000 UTC m=+29.613823794" watchObservedRunningTime="2026-07-16 21:50:29.518565609 +0000 UTC m=+29.614076926"
Jul 16 21:50:30.145381 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.145349    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wjrxm"]
Jul 16 21:50:30.145566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.145500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:30.145641 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:30.145620    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:30.150275 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.150253    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsh7v"]
Jul 16 21:50:30.150385 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.150373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:30.150482 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:30.150463    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:30.156406 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.156385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8twsk"]
Jul 16 21:50:30.156516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.156467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:30.156593 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:30.156558    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:30.493175 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.492991    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="cf6373df6040e28d3888986baf755e7fd80bfda80c7132622d976f20fe078e0a" exitCode=0
Jul 16 21:50:30.493175 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:30.493077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"cf6373df6040e28d3888986baf755e7fd80bfda80c7132622d976f20fe078e0a"}
Jul 16 21:50:31.313778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:31.313739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:31.313778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:31.313777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:31.313993 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:31.313913    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:31.314277 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:31.314247    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:32.314107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:32.314046    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:32.314603 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:32.314218    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:33.314216 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:33.314185    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:33.314936 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.314325    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:33.314936 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:33.314433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:33.314936 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.314637    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:33.978312 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:33.978268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:33.978507 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:33.978373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:33.978507 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.978426    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:33.978507 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.978486    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:33.978507 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.978506    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs podName:fdc51d15-a2f4-40e6-b973-11dd4094fe35 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.978486032 +0000 UTC m=+66.073997337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs") pod "network-metrics-daemon-jsh7v" (UID: "fdc51d15-a2f4-40e6-b973-11dd4094fe35") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:33.978720 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:33.978536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret podName:89297fd4-1e1b-42b3-aaec-182a16f6a4b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.978520848 +0000 UTC m=+66.074032147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret") pod "global-pull-secret-syncer-wjrxm" (UID: "89297fd4-1e1b-42b3-aaec-182a16f6a4b9") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:34.079001 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:34.078969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:34.079198 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:34.079168    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:34.079198 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:34.079202    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:34.079344 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:34.079216    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqpqb for pod openshift-network-diagnostics/network-check-target-8twsk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:34.079344 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:34.079283    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb podName:0cbea06d-d06c-4f8e-aad2-90a6af91b1e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.079262847 +0000 UTC m=+66.174774163 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqpqb" (UniqueName: "kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb") pod "network-check-target-8twsk" (UID: "0cbea06d-d06c-4f8e-aad2-90a6af91b1e4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:34.314059 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:34.313980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:34.314221 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:34.314107    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:35.314294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:35.314217    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:35.314294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:35.314263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:35.314681 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:35.314342    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jsh7v" podUID="fdc51d15-a2f4-40e6-b973-11dd4094fe35"
Jul 16 21:50:35.314681 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:35.314503    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wjrxm" podUID="89297fd4-1e1b-42b3-aaec-182a16f6a4b9"
Jul 16 21:50:35.508606 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:35.508577    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="da0997f7f7c2c176238f8861bf10ed99fe7a29b2c6ce6c67c97988cd8f27742b" exitCode=0
Jul 16 21:50:35.508772 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:35.508626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"da0997f7f7c2c176238f8861bf10ed99fe7a29b2c6ce6c67c97988cd8f27742b"}
Jul 16 21:50:36.314126 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.314089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:36.314328 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:36.314194    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8twsk" podUID="0cbea06d-d06c-4f8e-aad2-90a6af91b1e4"
Jul 16 21:50:36.513713 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.513674    2566 generic.go:356] "Generic (PLEG): container finished" podID="10098757-a07b-47c0-9786-61b5704881b0" containerID="639dbee5e43a23a7aed0be2898c2e5e9dfa52fb7c6924a05e71b3bec852c182f" exitCode=0
Jul 16 21:50:36.513897 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.513737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerDied","Data":"639dbee5e43a23a7aed0be2898c2e5e9dfa52fb7c6924a05e71b3bec852c182f"}
Jul 16 21:50:36.701228 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.701050    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-100.ec2.internal" event="NodeReady"
Jul 16 21:50:36.701378 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.701365    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:50:36.762503 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.762465    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:50:36.779970 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.779926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.781622 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.781597    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zq788"]
Jul 16 21:50:36.782569 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.782554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:50:36.786280 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.786255    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-47gk2\""
Jul 16 21:50:36.786865 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.786837    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:50:36.787073 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.787060    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:50:36.804081 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.804057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:36.807888 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.807854    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:50:36.808030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.808000    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:50:36.808182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.808157    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:50:36.808304 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.808157    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vqlsj\""
Jul 16 21:50:36.808455 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.808441    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c4cx2"]
Jul 16 21:50:36.813551 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.813533    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:50:36.838695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.838620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:50:36.838695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.838656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq788"]
Jul 16 21:50:36.838695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.838670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4cx2"]
Jul 16 21:50:36.838954 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.838764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:36.841971 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.841950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:50:36.842784 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.842763    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:50:36.843102 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.843088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g56tn\""
Jul 16 21:50:36.897749 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.897749 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.897966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.897966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897844    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.897966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.897966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35f0b32-6ae6-4250-b447-9c1ab70b5235-cert\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:36.898088 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.898088 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.897999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.898088 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.898039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6b9l\" (UniqueName: \"kubernetes.io/projected/c35f0b32-6ae6-4250-b447-9c1ab70b5235-kube-api-access-c6b9l\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:36.898088 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.898055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bnc\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.998935 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.998897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.998935 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.998944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.998962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35f0b32-6ae6-4250-b447-9c1ab70b5235-cert\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.998982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013a28d9-1af6-4734-a6bb-45ce6753a567-tmp-dir\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6b9l\" (UniqueName: \"kubernetes.io/projected/c35f0b32-6ae6-4250-b447-9c1ab70b5235-kube-api-access-c6b9l\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bnc\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:36.999187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgtl\" (UniqueName: \"kubernetes.io/projected/013a28d9-1af6-4734-a6bb-45ce6753a567-kube-api-access-sdgtl\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:36.999423    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:36.999446    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645c97ddf8-hctht: secret "image-registry-tls" not found
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:36.999499    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls podName:01fb8c89-4b40-433a-9d73-9baa307731fe nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.499481288 +0000 UTC m=+37.594992582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls") pod "image-registry-645c97ddf8-hctht" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe") : secret "image-registry-tls" not found
Jul 16 21:50:36.999519 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999853 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999441    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:36.999853 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/013a28d9-1af6-4734-a6bb-45ce6753a567-config-volume\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:36.999952 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:36.999874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.000148 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.000122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.003455 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.003433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.003559 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.003457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c35f0b32-6ae6-4250-b447-9c1ab70b5235-cert\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:37.003559 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.003436    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.010142 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.010121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.010440 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.010418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bnc\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.011150 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.011130    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6b9l\" (UniqueName: \"kubernetes.io/projected/c35f0b32-6ae6-4250-b447-9c1ab70b5235-kube-api-access-c6b9l\") pod \"ingress-canary-zq788\" (UID: \"c35f0b32-6ae6-4250-b447-9c1ab70b5235\") " pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:37.100778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.100675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/013a28d9-1af6-4734-a6bb-45ce6753a567-config-volume\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.100778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.100721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013a28d9-1af6-4734-a6bb-45ce6753a567-tmp-dir\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.100778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.100766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.101065 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.100784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgtl\" (UniqueName: \"kubernetes.io/projected/013a28d9-1af6-4734-a6bb-45ce6753a567-kube-api-access-sdgtl\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.101065 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.100942    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.101065 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.101002    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls podName:013a28d9-1af6-4734-a6bb-45ce6753a567 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.600985507 +0000 UTC m=+37.696496804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls") pod "dns-default-c4cx2" (UID: "013a28d9-1af6-4734-a6bb-45ce6753a567") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.101225 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.101137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013a28d9-1af6-4734-a6bb-45ce6753a567-tmp-dir\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.101329 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.101313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/013a28d9-1af6-4734-a6bb-45ce6753a567-config-volume\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.111765 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.111744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgtl\" (UniqueName: \"kubernetes.io/projected/013a28d9-1af6-4734-a6bb-45ce6753a567-kube-api-access-sdgtl\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.113647 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.113627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq788"
Jul 16 21:50:37.313731 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.313699    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:50:37.313895 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.313710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:50:37.316741 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.316710    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:37.317110 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.316749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn8m8\""
Jul 16 21:50:37.317110 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.316862    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:37.322554 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.322532    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq788"]
Jul 16 21:50:37.327117 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:37.327092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35f0b32_6ae6_4250_b447_9c1ab70b5235.slice/crio-001f69a35a6925f509d374547b12f09adedf8c40653c8b718316ff2ac167ab7a WatchSource:0}: Error finding container 001f69a35a6925f509d374547b12f09adedf8c40653c8b718316ff2ac167ab7a: Status 404 returned error can't find the container with id 001f69a35a6925f509d374547b12f09adedf8c40653c8b718316ff2ac167ab7a
Jul 16 21:50:37.476698 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.476663    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p9b47"]
Jul 16 21:50:37.498496 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.498469    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.501204 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.501178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q652x\""
Jul 16 21:50:37.501450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.501421    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:50:37.501564 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.501541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:50:37.501843 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.501642    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:50:37.501943 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.501852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:50:37.502248 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.502226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9b47"]
Jul 16 21:50:37.504035 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.504016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:37.504256 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.504174    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:37.504256 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.504199    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645c97ddf8-hctht: secret "image-registry-tls" not found
Jul 16 21:50:37.504415 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.504261    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls podName:01fb8c89-4b40-433a-9d73-9baa307731fe nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.504240864 +0000 UTC m=+38.599752162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls") pod "image-registry-645c97ddf8-hctht" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe") : secret "image-registry-tls" not found
Jul 16 21:50:37.519681 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.519643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c899z" event={"ID":"10098757-a07b-47c0-9786-61b5704881b0","Type":"ContainerStarted","Data":"b8dd8645f8ab8b7e83f35b4ed3bb9f418336517b25b19a183d5dde268ad5cc1f"}
Jul 16 21:50:37.520809 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.520784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq788" event={"ID":"c35f0b32-6ae6-4250-b447-9c1ab70b5235","Type":"ContainerStarted","Data":"001f69a35a6925f509d374547b12f09adedf8c40653c8b718316ff2ac167ab7a"}
Jul 16 21:50:37.548877 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.548792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c899z" podStartSLOduration=6.376319181 podStartE2EDuration="36.548775022s" totalImagesPullingTime="30.172455841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:37.547624563 +0000 UTC m=+37.643135889" watchObservedRunningTime="2026-07-16 21:50:37.548775022 +0000 UTC m=+37.644286342"
Jul 16 21:50:37.604896 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.604856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7e0641-b4ef-4104-8a71-b9795f4c6340-crio-socket\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.605044 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.605010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lfw\" (UniqueName: \"kubernetes.io/projected/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-api-access-n5lfw\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.605155 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.605053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.605155 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.605084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.605155 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.605137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:37.605316 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.605183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7e0641-b4ef-4104-8a71-b9795f4c6340-data-volume\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.605316 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.605301    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.605411 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.605358    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls podName:013a28d9-1af6-4734-a6bb-45ce6753a567 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.605339017 +0000 UTC m=+38.700850312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls") pod "dns-default-c4cx2" (UID: "013a28d9-1af6-4734-a6bb-45ce6753a567") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.705538 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7e0641-b4ef-4104-8a71-b9795f4c6340-data-volume\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705711 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705552    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7e0641-b4ef-4104-8a71-b9795f4c6340-crio-socket\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705711 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lfw\" (UniqueName: \"kubernetes.io/projected/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-api-access-n5lfw\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705711 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705711 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.705741    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a7e0641-b4ef-4104-8a71-b9795f4c6340-crio-socket\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.705743    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.705846    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.205807041 +0000 UTC m=+38.301318350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:37.705875    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.20586194 +0000 UTC m=+38.301373266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:37.705937 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.705893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a7e0641-b4ef-4104-8a71-b9795f4c6340-data-volume\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:37.715539 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:37.715510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lfw\" (UniqueName: \"kubernetes.io/projected/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-api-access-n5lfw\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:38.210938 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.210899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:38.210938 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.210944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:38.211153 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.211038    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:38.211153 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.211065    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:38.211153 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.211117    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:39.211099177 +0000 UTC m=+39.306610472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:38.211153 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.211132    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:39.211125974 +0000 UTC m=+39.306637269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:38.314581 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.314338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:50:38.317346 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.317325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qpnd8\""
Jul 16 21:50:38.317734 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.317373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:38.318117 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.318094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:38.514753 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.514666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:38.514912 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.514810    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:38.514912 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.514847    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645c97ddf8-hctht: secret "image-registry-tls" not found
Jul 16 21:50:38.514997 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.514922    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls podName:01fb8c89-4b40-433a-9d73-9baa307731fe nodeName:}" failed. No retries permitted until 2026-07-16 21:50:40.51490566 +0000 UTC m=+40.610416975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls") pod "image-registry-645c97ddf8-hctht" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe") : secret "image-registry-tls" not found
Jul 16 21:50:38.615811 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:38.615773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:38.616001 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.615930    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:38.616066 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:38.616000    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls podName:013a28d9-1af6-4734-a6bb-45ce6753a567 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:40.615978983 +0000 UTC m=+40.711490285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls") pod "dns-default-c4cx2" (UID: "013a28d9-1af6-4734-a6bb-45ce6753a567") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:39.222726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:39.222686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:39.222726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:39.222729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:39.223025 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:39.222826    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:50:39.223025 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:39.222896    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.22287931 +0000 UTC m=+41.318390607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : configmap "kube-rbac-proxy" not found
Jul 16 21:50:39.223025 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:39.222935    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:39.223025 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:39.223004    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.222989993 +0000 UTC m=+41.318501291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:40.254380 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:40.254354    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9mc6r_6ea52686-3288-4855-aef7-2ba5ac429dbf/dns-node-resolver/0.log"
Jul 16 21:50:40.529594 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:40.529510    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq788" event={"ID":"c35f0b32-6ae6-4250-b447-9c1ab70b5235","Type":"ContainerStarted","Data":"e3bca526d2a53eb3f0ba906ba81a7b1581ece81123c063fe3a57d7b6f3b2fc69"}
Jul 16 21:50:40.534012 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:40.533986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:40.534140 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:40.534111    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:40.534140 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:40.534125    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645c97ddf8-hctht: secret "image-registry-tls" not found
Jul 16 21:50:40.534224 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:40.534172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls podName:01fb8c89-4b40-433a-9d73-9baa307731fe nodeName:}" failed. No retries permitted until 2026-07-16 21:50:44.534160046 +0000 UTC m=+44.629671341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls") pod "image-registry-645c97ddf8-hctht" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe") : secret "image-registry-tls" not found
Jul 16 21:50:40.556023 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:40.555982    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zq788" podStartSLOduration=2.2651715660000002 podStartE2EDuration="4.555970418s" totalImagesPullingTime="2.290798852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:40.55448347 +0000 UTC m=+40.649994787" watchObservedRunningTime="2026-07-16 21:50:40.555970418 +0000 UTC m=+40.651481714"
Jul 16 21:50:40.634807 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:40.634771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:40.634967 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:40.634946    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:40.635035 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:40.635017    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls podName:013a28d9-1af6-4734-a6bb-45ce6753a567 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:44.634999653 +0000 UTC m=+44.730510955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls") pod "dns-default-c4cx2" (UID: "013a28d9-1af6-4734-a6bb-45ce6753a567") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:41.238808 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:41.238772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:41.238808 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:41.238809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:41.239080 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:41.238951    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:41.239080 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:41.239000    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.238987168 +0000 UTC m=+45.334498463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:41.239422 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:41.239394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a7e0641-b4ef-4104-8a71-b9795f4c6340-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:41.256168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:41.256146    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fs66g_e1304cb1-326c-4c14-8464-201068718829/node-ca/0.log"
Jul 16 21:50:42.254830 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:42.254779    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq788_c35f0b32-6ae6-4250-b447-9c1ab70b5235/serve-healthcheck-canary/0.log"
Jul 16 21:50:44.562410 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:44.562374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:44.562962 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:44.562944    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:44.562962 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:44.562964    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-645c97ddf8-hctht: secret "image-registry-tls" not found
Jul 16 21:50:44.563054 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:44.563013    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls podName:01fb8c89-4b40-433a-9d73-9baa307731fe nodeName:}" failed. No retries permitted until 2026-07-16 21:50:52.56299921 +0000 UTC m=+52.658510504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls") pod "image-registry-645c97ddf8-hctht" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe") : secret "image-registry-tls" not found
Jul 16 21:50:44.662741 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:44.662704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:44.662913 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:44.662876    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:44.662956 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:44.662936    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls podName:013a28d9-1af6-4734-a6bb-45ce6753a567 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:52.662918501 +0000 UTC m=+52.758429798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls") pod "dns-default-c4cx2" (UID: "013a28d9-1af6-4734-a6bb-45ce6753a567") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:45.267450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:45.267418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:45.267623 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:45.267581    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:45.267663 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:50:45.267648    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls podName:7a7e0641-b4ef-4104-8a71-b9795f4c6340 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:53.26763095 +0000 UTC m=+53.363142246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p9b47" (UID: "7a7e0641-b4ef-4104-8a71-b9795f4c6340") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:50:52.625668 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.625625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:52.628608 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.628585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"image-registry-645c97ddf8-hctht\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") " pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:52.689509 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.689472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:52.725993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.725962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:52.729053 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.729023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/013a28d9-1af6-4734-a6bb-45ce6753a567-metrics-tls\") pod \"dns-default-c4cx2\" (UID: \"013a28d9-1af6-4734-a6bb-45ce6753a567\") " pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:52.747990 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.747954    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:52.839634 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.839606    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:50:52.842679 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:52.842637    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fb8c89_4b40_433a_9d73_9baa307731fe.slice/crio-aad9602eb52d6537115ef318289b6dd923e08c0c564e0001337ddf097cda8c29 WatchSource:0}: Error finding container aad9602eb52d6537115ef318289b6dd923e08c0c564e0001337ddf097cda8c29: Status 404 returned error can't find the container with id aad9602eb52d6537115ef318289b6dd923e08c0c564e0001337ddf097cda8c29
Jul 16 21:50:52.900255 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:52.900223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c4cx2"]
Jul 16 21:50:52.904799 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:52.904765    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013a28d9_1af6_4734_a6bb_45ce6753a567.slice/crio-be617400ecd6e758f2da11ffe35ac18654f32aaa1392cab144d356b28062c0e0 WatchSource:0}: Error finding container be617400ecd6e758f2da11ffe35ac18654f32aaa1392cab144d356b28062c0e0: Status 404 returned error can't find the container with id be617400ecd6e758f2da11ffe35ac18654f32aaa1392cab144d356b28062c0e0
Jul 16 21:50:53.330954 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.330905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:53.333204 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.333177    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a7e0641-b4ef-4104-8a71-b9795f4c6340-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p9b47\" (UID: \"7a7e0641-b4ef-4104-8a71-b9795f4c6340\") " pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:53.408886 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.408853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p9b47"
Jul 16 21:50:53.546368 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.546331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p9b47"]
Jul 16 21:50:53.549739 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:53.549702    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7e0641_b4ef_4104_8a71_b9795f4c6340.slice/crio-4e444cc470dce108443e2a0319c8aa0e0182248e2b488a566d9833cee543f5ca WatchSource:0}: Error finding container 4e444cc470dce108443e2a0319c8aa0e0182248e2b488a566d9833cee543f5ca: Status 404 returned error can't find the container with id 4e444cc470dce108443e2a0319c8aa0e0182248e2b488a566d9833cee543f5ca
Jul 16 21:50:53.562360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.562307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" event={"ID":"01fb8c89-4b40-433a-9d73-9baa307731fe","Type":"ContainerStarted","Data":"c68c2a807fac682beeb8ed0233d77505b7c23e2c6236e61f02138e92fb0064b5"}
Jul 16 21:50:53.562490 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.562385    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" event={"ID":"01fb8c89-4b40-433a-9d73-9baa307731fe","Type":"ContainerStarted","Data":"aad9602eb52d6537115ef318289b6dd923e08c0c564e0001337ddf097cda8c29"}
Jul 16 21:50:53.562490 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.562441    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:50:53.563922 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.563880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9b47" event={"ID":"7a7e0641-b4ef-4104-8a71-b9795f4c6340","Type":"ContainerStarted","Data":"4e444cc470dce108443e2a0319c8aa0e0182248e2b488a566d9833cee543f5ca"}
Jul 16 21:50:53.565141 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.565109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4cx2" event={"ID":"013a28d9-1af6-4734-a6bb-45ce6753a567","Type":"ContainerStarted","Data":"be617400ecd6e758f2da11ffe35ac18654f32aaa1392cab144d356b28062c0e0"}
Jul 16 21:50:53.585800 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:53.585669    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" podStartSLOduration=24.585650525 podStartE2EDuration="24.585650525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:53.584990229 +0000 UTC m=+53.680501549" watchObservedRunningTime="2026-07-16 21:50:53.585650525 +0000 UTC m=+53.681161841"
Jul 16 21:50:55.574687 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:55.574651    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4cx2" event={"ID":"013a28d9-1af6-4734-a6bb-45ce6753a567","Type":"ContainerStarted","Data":"c702f51220810c4fb3a2d3dd5794f8053bfba663cc3cb1ed1762d67e556ca593"}
Jul 16 21:50:55.575135 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:55.574747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c4cx2" event={"ID":"013a28d9-1af6-4734-a6bb-45ce6753a567","Type":"ContainerStarted","Data":"6428db240b03014b2f2c8cd166335384e1ef224d3c6a27cdf49524448fa95201"}
Jul 16 21:50:55.575135 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:55.574781    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:50:55.605317 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:55.605262    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c4cx2" podStartSLOduration=17.668589917 podStartE2EDuration="19.605244862s" totalImagesPullingTime="1.936654945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:55.604539738 +0000 UTC m=+55.700051055" watchObservedRunningTime="2026-07-16 21:50:55.605244862 +0000 UTC m=+55.700756178"
Jul 16 21:50:58.444245 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.444206    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"]
Jul 16 21:50:58.449971 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.449949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.462353 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.462318    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 21:50:58.463470 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.463449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.463593 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.463449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.464660 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.464631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 21:50:58.471472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.470018    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:50:58.471472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.470115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.471472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.470172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-tmp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.471472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.470191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvrdp\" (UniqueName: \"kubernetes.io/projected/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-kube-api-access-fvrdp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.471472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.471051    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"]
Jul 16 21:50:58.475437 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.475419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.482486 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.481936    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v45dl\""
Jul 16 21:50:58.482486 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.481953    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 16 21:50:58.482486 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.482199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 16 21:50:58.492694 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.492666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"]
Jul 16 21:50:58.515263 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.515222    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"]
Jul 16 21:50:58.566390 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.565582    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"]
Jul 16 21:50:58.568992 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.568966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.570768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5j8\" (UniqueName: \"kubernetes.io/projected/447ce3db-da02-4e22-a2e2-49ecd27edce3-kube-api-access-zk5j8\") pod \"network-check-source-7bc5b6bf85-zc2c6\" (UID: \"447ce3db-da02-4e22-a2e2-49ecd27edce3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.570878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.570924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.570963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-tmp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.570981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvrdp\" (UniqueName: \"kubernetes.io/projected/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-kube-api-access-fvrdp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.571021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.571448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.571406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-tmp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.572628 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.572604    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5vfcj\""
Jul 16 21:50:58.574037 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.574010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-klusterlet-config\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.586795 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.586765    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fcb87f568-6gz4t"]
Jul 16 21:50:58.590585 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.590565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.610411 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.610377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"]
Jul 16 21:50:58.613460 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.613437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvrdp\" (UniqueName: \"kubernetes.io/projected/c88e6bd1-1f0b-4129-9bf3-c52573066a6c-kube-api-access-fvrdp\") pod \"klusterlet-addon-workmgr-686d9678-jlvsr\" (UID: \"c88e6bd1-1f0b-4129-9bf3-c52573066a6c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.634459 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.634422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fcb87f568-6gz4t"]
Jul 16 21:50:58.641963 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.641932    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-z98vj"]
Jul 16 21:50:58.645366 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.645341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:50:58.648491 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.648461    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.648845 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.648642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.648845 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.648713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-p6bbf\""
Jul 16 21:50:58.664794 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.664765    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-z98vj"]
Jul 16 21:50:58.672151 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-tls\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672267 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.672267 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672192    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-trusted-ca\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672338 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qk8\" (UniqueName: \"kubernetes.io/projected/bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687-kube-api-access-64qk8\") pod \"downloads-5bc67dcd46-z98vj\" (UID: \"bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687\") " pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:50:58.672370 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5j8\" (UniqueName: \"kubernetes.io/projected/447ce3db-da02-4e22-a2e2-49ecd27edce3-kube-api-access-zk5j8\") pod \"network-check-source-7bc5b6bf85-zc2c6\" (UID: \"447ce3db-da02-4e22-a2e2-49ecd27edce3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"
Jul 16 21:50:58.672409 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb625f60-4985-418b-ac4e-241c97dd45c6-ca-trust-extracted\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672454 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sndpm\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-kube-api-access-sndpm\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672454 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-certificates\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672532 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-installation-pull-secrets\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672532 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-image-registry-private-configuration\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672605 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.672605 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-bound-sa-token\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.672923 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.672902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.674987 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.674970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9fa92db6-85f8-4b66-aa03-7c8fa2457c73-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-krmrx\" (UID: \"9fa92db6-85f8-4b66-aa03-7c8fa2457c73\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.685510 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.685484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5j8\" (UniqueName: \"kubernetes.io/projected/447ce3db-da02-4e22-a2e2-49ecd27edce3-kube-api-access-zk5j8\") pod \"network-check-source-7bc5b6bf85-zc2c6\" (UID: \"447ce3db-da02-4e22-a2e2-49ecd27edce3\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"
Jul 16 21:50:58.763665 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.763628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:50:58.772946 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.772917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-tls\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773060 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.772957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-trusted-ca\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773060 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.772991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qk8\" (UniqueName: \"kubernetes.io/projected/bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687-kube-api-access-64qk8\") pod \"downloads-5bc67dcd46-z98vj\" (UID: \"bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687\") " pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:50:58.773060 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb625f60-4985-418b-ac4e-241c97dd45c6-ca-trust-extracted\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773060 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sndpm\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-kube-api-access-sndpm\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-certificates\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-installation-pull-secrets\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-image-registry-private-configuration\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-bound-sa-token\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.773570 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.773541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb625f60-4985-418b-ac4e-241c97dd45c6-ca-trust-extracted\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.774104 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.774084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-trusted-ca\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.774277 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.774257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-certificates\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.775734 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.775712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-registry-tls\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.775834 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.775714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-installation-pull-secrets\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.775896 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.775868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb625f60-4985-418b-ac4e-241c97dd45c6-image-registry-private-configuration\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.786443 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.786276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sndpm\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-kube-api-access-sndpm\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.787789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.787772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"
Jul 16 21:50:58.788211 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.788193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qk8\" (UniqueName: \"kubernetes.io/projected/bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687-kube-api-access-64qk8\") pod \"downloads-5bc67dcd46-z98vj\" (UID: \"bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687\") " pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:50:58.788858 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.788810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb625f60-4985-418b-ac4e-241c97dd45c6-bound-sa-token\") pod \"image-registry-5fcb87f568-6gz4t\" (UID: \"eb625f60-4985-418b-ac4e-241c97dd45c6\") " pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.886221 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.885993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"
Jul 16 21:50:58.900290 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.900261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:58.910558 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.910528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"]
Jul 16 21:50:58.943536 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.943495    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx"]
Jul 16 21:50:58.957531 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:58.957507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:50:59.077548 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:59.077461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88e6bd1_1f0b_4129_9bf3_c52573066a6c.slice/crio-3fe2317025b5c14461d23f153f89645c82c5d1ed6af2369b3f3e11fe0075377f WatchSource:0}: Error finding container 3fe2317025b5c14461d23f153f89645c82c5d1ed6af2369b3f3e11fe0075377f: Status 404 returned error can't find the container with id 3fe2317025b5c14461d23f153f89645c82c5d1ed6af2369b3f3e11fe0075377f
Jul 16 21:50:59.077979 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:59.077957    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa92db6_85f8_4b66_aa03_7c8fa2457c73.slice/crio-6af6c45b0f02479463f2601c62dae30d1388eaa5b54d9fb1593ffdcb18f8758d WatchSource:0}: Error finding container 6af6c45b0f02479463f2601c62dae30d1388eaa5b54d9fb1593ffdcb18f8758d: Status 404 returned error can't find the container with id 6af6c45b0f02479463f2601c62dae30d1388eaa5b54d9fb1593ffdcb18f8758d
Jul 16 21:50:59.260425 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.260395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fcb87f568-6gz4t"]
Jul 16 21:50:59.264473 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:59.264442    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb625f60_4985_418b_ac4e_241c97dd45c6.slice/crio-0e54d05a0cfc19bdda878164c91406d1c3155baa3e26b5f15b1edfe43b75b14a WatchSource:0}: Error finding container 0e54d05a0cfc19bdda878164c91406d1c3155baa3e26b5f15b1edfe43b75b14a: Status 404 returned error can't find the container with id 0e54d05a0cfc19bdda878164c91406d1c3155baa3e26b5f15b1edfe43b75b14a
Jul 16 21:50:59.492858 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.492827    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-z98vj"]
Jul 16 21:50:59.496737 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:59.496710    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa7f7d4_d2e4_4dfc_93f6_fedb5d560687.slice/crio-78a8333d53a3b681c96abac9630c045a3a68c0c8be76f57a17e9fb617d499836 WatchSource:0}: Error finding container 78a8333d53a3b681c96abac9630c045a3a68c0c8be76f57a17e9fb617d499836: Status 404 returned error can't find the container with id 78a8333d53a3b681c96abac9630c045a3a68c0c8be76f57a17e9fb617d499836
Jul 16 21:50:59.497967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.497942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6"]
Jul 16 21:50:59.502071 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:50:59.502048    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447ce3db_da02_4e22_a2e2_49ecd27edce3.slice/crio-51e4c38d0bafa87dc4531b175cd39c9c23dd35523d6843ec71ccc09bd5725bbe WatchSource:0}: Error finding container 51e4c38d0bafa87dc4531b175cd39c9c23dd35523d6843ec71ccc09bd5725bbe: Status 404 returned error can't find the container with id 51e4c38d0bafa87dc4531b175cd39c9c23dd35523d6843ec71ccc09bd5725bbe
Jul 16 21:50:59.596509 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.596365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-z98vj" event={"ID":"bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687","Type":"ContainerStarted","Data":"78a8333d53a3b681c96abac9630c045a3a68c0c8be76f57a17e9fb617d499836"}
Jul 16 21:50:59.598570 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.598540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx" event={"ID":"9fa92db6-85f8-4b66-aa03-7c8fa2457c73","Type":"ContainerStarted","Data":"6af6c45b0f02479463f2601c62dae30d1388eaa5b54d9fb1593ffdcb18f8758d"}
Jul 16 21:50:59.601838 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.601784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr" event={"ID":"c88e6bd1-1f0b-4129-9bf3-c52573066a6c","Type":"ContainerStarted","Data":"3fe2317025b5c14461d23f153f89645c82c5d1ed6af2369b3f3e11fe0075377f"}
Jul 16 21:50:59.604022 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.603993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6" event={"ID":"447ce3db-da02-4e22-a2e2-49ecd27edce3","Type":"ContainerStarted","Data":"51e4c38d0bafa87dc4531b175cd39c9c23dd35523d6843ec71ccc09bd5725bbe"}
Jul 16 21:50:59.605784 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.605762    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t" event={"ID":"eb625f60-4985-418b-ac4e-241c97dd45c6","Type":"ContainerStarted","Data":"01b7feca77adccd0bb96897f522583a31d05e0df867148a88e99a9e6ed093b64"}
Jul 16 21:50:59.605908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.605804    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t" event={"ID":"eb625f60-4985-418b-ac4e-241c97dd45c6","Type":"ContainerStarted","Data":"0e54d05a0cfc19bdda878164c91406d1c3155baa3e26b5f15b1edfe43b75b14a"}
Jul 16 21:50:59.605908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.605868    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:50:59.608739 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.608690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9b47" event={"ID":"7a7e0641-b4ef-4104-8a71-b9795f4c6340","Type":"ContainerStarted","Data":"a8bd7c26408a6c94747c02161a1fb638cfd7f1cb94e0433fdc60459e8b73e44e"}
Jul 16 21:50:59.632302 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:50:59.632201    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t" podStartSLOduration=1.632184438 podStartE2EDuration="1.632184438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:59.631790681 +0000 UTC m=+59.727301999" watchObservedRunningTime="2026-07-16 21:50:59.632184438 +0000 UTC m=+59.727695755"
Jul 16 21:51:01.530289 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:01.528915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jndh2"
Jul 16 21:51:01.626698 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:01.625953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx" event={"ID":"9fa92db6-85f8-4b66-aa03-7c8fa2457c73","Type":"ContainerStarted","Data":"0e233727e7fe60ccdd5acc661be6bb3d121604bd9d89bb7b3d7c61fd4d6df774"}
Jul 16 21:51:05.581244 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:05.581134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c4cx2"
Jul 16 21:51:05.638639 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:05.638582    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-krmrx" podStartSLOduration=5.924657762 podStartE2EDuration="7.638565217s" totalImagesPullingTime="1.713907455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:01.653436362 +0000 UTC m=+61.748947681" watchObservedRunningTime="2026-07-16 21:51:05.638565217 +0000 UTC m=+65.734076533"
Jul 16 21:51:06.033092 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.033052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:51:06.033272 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.033128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:51:06.037293 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.037262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:51:06.037519 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.037503    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:51:06.046178 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.046140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89297fd4-1e1b-42b3-aaec-182a16f6a4b9-original-pull-secret\") pod \"global-pull-secret-syncer-wjrxm\" (UID: \"89297fd4-1e1b-42b3-aaec-182a16f6a4b9\") " pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:51:06.046515 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.046481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc51d15-a2f4-40e6-b973-11dd4094fe35-metrics-certs\") pod \"network-metrics-daemon-jsh7v\" (UID: \"fdc51d15-a2f4-40e6-b973-11dd4094fe35\") " pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:51:06.129894 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.129851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wjrxm"
Jul 16 21:51:06.131929 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.131897    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn8m8\""
Jul 16 21:51:06.134306 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.134276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:51:06.134839 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.134759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jsh7v"
Jul 16 21:51:06.137593 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.137569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpqb\" (UniqueName: \"kubernetes.io/projected/0cbea06d-d06c-4f8e-aad2-90a6af91b1e4-kube-api-access-zqpqb\") pod \"network-check-target-8twsk\" (UID: \"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4\") " pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:51:06.230342 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.230311    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qpnd8\""
Jul 16 21:51:06.236615 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:06.236577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:51:07.596205 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.596175    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:51:07.599608 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.599582    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.607081 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.607054    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lc2gx\""
Jul 16 21:51:07.607211 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.607130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:51:07.608114 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.608090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:51:07.608360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.608339    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:51:07.608596 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.608560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:51:07.609918 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.609723    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:51:07.636216 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.636180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:51:07.648117 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.648325 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.648325 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.648325 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.648550 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.648550 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.648383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8drr\" (UniqueName: \"kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8drr\" (UniqueName: \"kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.751750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.752481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.752706    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.753789 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.753060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.788337 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.788293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.788796 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.788767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.795574 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.795118    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8drr\" (UniqueName: \"kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr\") pod \"console-5c65fb7f45-pckhx\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") " pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.894628 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.894553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wjrxm"]
Jul 16 21:51:07.900209 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:07.900164    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89297fd4_1e1b_42b3_aaec_182a16f6a4b9.slice/crio-07ae4406918062d5f0ed214ad1aba94845b5bf6e351a67a31445b7e9e6027ac6 WatchSource:0}: Error finding container 07ae4406918062d5f0ed214ad1aba94845b5bf6e351a67a31445b7e9e6027ac6: Status 404 returned error can't find the container with id 07ae4406918062d5f0ed214ad1aba94845b5bf6e351a67a31445b7e9e6027ac6
Jul 16 21:51:07.910890 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.910863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:07.914758 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.914666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8twsk"]
Jul 16 21:51:07.918717 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:07.918682    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbea06d_d06c_4f8e_aad2_90a6af91b1e4.slice/crio-f9f82c1081cdbd62c0dd53507c34e1b8f5a1c96350d94f7089feda8a6fd58e6b WatchSource:0}: Error finding container f9f82c1081cdbd62c0dd53507c34e1b8f5a1c96350d94f7089feda8a6fd58e6b: Status 404 returned error can't find the container with id f9f82c1081cdbd62c0dd53507c34e1b8f5a1c96350d94f7089feda8a6fd58e6b
Jul 16 21:51:07.926614 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:07.926582    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jsh7v"]
Jul 16 21:51:07.930469 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:07.930428    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc51d15_a2f4_40e6_b973_11dd4094fe35.slice/crio-36b2c72d374d36866fe3b0125ccee5ebecbfbbe081677894fbfb6707b92b4bf0 WatchSource:0}: Error finding container 36b2c72d374d36866fe3b0125ccee5ebecbfbbe081677894fbfb6707b92b4bf0: Status 404 returned error can't find the container with id 36b2c72d374d36866fe3b0125ccee5ebecbfbbe081677894fbfb6707b92b4bf0
Jul 16 21:51:08.080258 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.080211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:51:08.084856 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:08.084805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f03ee34_31ee_46ed_9bbe_67e69da979ce.slice/crio-aa90df637585e5e3769881663375abf0ecf9fd068e5dfdb3592d862345083d11 WatchSource:0}: Error finding container aa90df637585e5e3769881663375abf0ecf9fd068e5dfdb3592d862345083d11: Status 404 returned error can't find the container with id aa90df637585e5e3769881663375abf0ecf9fd068e5dfdb3592d862345083d11
Jul 16 21:51:08.477653 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.477350    2566 patch_prober.go:28] interesting pod/image-registry-645c97ddf8-hctht container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 21:51:08.477653 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.477413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:51:08.658139 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.658096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9b47" event={"ID":"7a7e0641-b4ef-4104-8a71-b9795f4c6340","Type":"ContainerStarted","Data":"1c28f1807ffad1ff35b069a2ccbc23ad2719057cbc10a90fab6704b5c042c36e"}
Jul 16 21:51:08.660695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.659874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8twsk" event={"ID":"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4","Type":"ContainerStarted","Data":"91064dc85f0bfac2c095eb34cd35830b274a210fae15df5df0ea0c3a4a2cbc8b"}
Jul 16 21:51:08.660695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.659919    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8twsk" event={"ID":"0cbea06d-d06c-4f8e-aad2-90a6af91b1e4","Type":"ContainerStarted","Data":"f9f82c1081cdbd62c0dd53507c34e1b8f5a1c96350d94f7089feda8a6fd58e6b"}
Jul 16 21:51:08.660695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.660612    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:51:08.665240 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.663066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr" event={"ID":"c88e6bd1-1f0b-4129-9bf3-c52573066a6c","Type":"ContainerStarted","Data":"a092ec9dc148b7878f1fc4926c22bc99e8da01f8cbe04ca79959f3e5d3d1d323"}
Jul 16 21:51:08.665240 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.665089    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:51:08.667396 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.667345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsh7v" event={"ID":"fdc51d15-a2f4-40e6-b973-11dd4094fe35","Type":"ContainerStarted","Data":"36b2c72d374d36866fe3b0125ccee5ebecbfbbe081677894fbfb6707b92b4bf0"}
Jul 16 21:51:08.667804 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.667752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr"
Jul 16 21:51:08.669873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.669545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6" event={"ID":"447ce3db-da02-4e22-a2e2-49ecd27edce3","Type":"ContainerStarted","Data":"3dcb9b28680d32bd09b4a853f348b2319186db36a296d5648bed9ca0aa4f588a"}
Jul 16 21:51:08.672401 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.672377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c65fb7f45-pckhx" event={"ID":"3f03ee34-31ee-46ed-9bbe-67e69da979ce","Type":"ContainerStarted","Data":"aa90df637585e5e3769881663375abf0ecf9fd068e5dfdb3592d862345083d11"}
Jul 16 21:51:08.674379 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.674321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wjrxm" event={"ID":"89297fd4-1e1b-42b3-aaec-182a16f6a4b9","Type":"ContainerStarted","Data":"07ae4406918062d5f0ed214ad1aba94845b5bf6e351a67a31445b7e9e6027ac6"}
Jul 16 21:51:08.693428 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.692950    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8twsk" podStartSLOduration=67.692929049 podStartE2EDuration="1m7.692929049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:08.691621168 +0000 UTC m=+68.787132486" watchObservedRunningTime="2026-07-16 21:51:08.692929049 +0000 UTC m=+68.788440367"
Jul 16 21:51:08.732215 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.732032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-686d9678-jlvsr" podStartSLOduration=2.16421 podStartE2EDuration="10.732007824s" totalImagesPullingTime="8.567797824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:08.729931728 +0000 UTC m=+68.825443051" watchObservedRunningTime="2026-07-16 21:51:08.732007824 +0000 UTC m=+68.827519143"
Jul 16 21:51:08.763743 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.763253    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-zc2c6" podStartSLOduration=2.598752545 podStartE2EDuration="10.763228971s" totalImagesPullingTime="8.164476426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:08.761547594 +0000 UTC m=+68.857058912" watchObservedRunningTime="2026-07-16 21:51:08.763228971 +0000 UTC m=+68.858740290"
Jul 16 21:51:08.802855 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.801632    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"]
Jul 16 21:51:08.807651 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.807437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:08.813138 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.813107    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"]
Jul 16 21:51:08.815689 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.815352    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 21:51:08.826130 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.826108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rq5nw\""
Jul 16 21:51:08.868989 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.868951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8fgw4\" (UID: \"14fec963-1510-4b5b-be31-706b04c39634\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:08.969631 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:08.969584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8fgw4\" (UID: \"14fec963-1510-4b5b-be31-706b04c39634\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:08.969860 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:08.969765    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 21:51:08.969938 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:08.969866    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates podName:14fec963-1510-4b5b-be31-706b04c39634 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:09.469842526 +0000 UTC m=+69.565353823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-8fgw4" (UID: "14fec963-1510-4b5b-be31-706b04c39634") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 21:51:09.474538 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:09.474500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8fgw4\" (UID: \"14fec963-1510-4b5b-be31-706b04c39634\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:09.474731 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:09.474667    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 21:51:09.474800 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:09.474743    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates podName:14fec963-1510-4b5b-be31-706b04c39634 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:10.474721168 +0000 UTC m=+70.570232468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-8fgw4" (UID: "14fec963-1510-4b5b-be31-706b04c39634") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 16 21:51:09.687738 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:09.687277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsh7v" event={"ID":"fdc51d15-a2f4-40e6-b973-11dd4094fe35","Type":"ContainerStarted","Data":"b36cf1c36cf07f913370b275769b37ad1ff5b58dfc212eaf71c3387ed4fe8014"}
Jul 16 21:51:09.687738 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:09.687331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jsh7v" event={"ID":"fdc51d15-a2f4-40e6-b973-11dd4094fe35","Type":"ContainerStarted","Data":"a764b419faa4504624227068deb7961f3e506fabff1bf8c525cdf8a1b7020959"}
Jul 16 21:51:09.711798 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:09.711622    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jsh7v" podStartSLOduration=67.386117913 podStartE2EDuration="1m8.711600466s" totalImagesPullingTime="1.325482553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:09.710886549 +0000 UTC m=+69.806397871" watchObservedRunningTime="2026-07-16 21:51:09.711600466 +0000 UTC m=+69.807111786"
Jul 16 21:51:10.486038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:10.485997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8fgw4\" (UID: \"14fec963-1510-4b5b-be31-706b04c39634\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:10.500726 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:10.500691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/14fec963-1510-4b5b-be31-706b04c39634-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8fgw4\" (UID: \"14fec963-1510-4b5b-be31-706b04c39634\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:10.629972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:10.629917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:10.838251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:10.838221    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"]
Jul 16 21:51:10.846870 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:10.841072    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fec963_1510_4b5b_be31_706b04c39634.slice/crio-964beec66770cd95bfc2d8ed0aba92675283fe638f0547a78a43b8988dd8838b WatchSource:0}: Error finding container 964beec66770cd95bfc2d8ed0aba92675283fe638f0547a78a43b8988dd8838b: Status 404 returned error can't find the container with id 964beec66770cd95bfc2d8ed0aba92675283fe638f0547a78a43b8988dd8838b
Jul 16 21:51:11.550792 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.550753    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:51:11.559661 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.559590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.576436 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.576357    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:51:11.577663 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.577528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:51:11.592540 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592540 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592771 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592771 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592771 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdd7\" (UniqueName: \"kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592771 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.592771 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.592671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694326 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.693459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdd7\" (UniqueName: \"kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694716 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.694716 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.695062 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.694252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.696149 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.695783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.696440 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.696417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.696792 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.696770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.699475 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.699447    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.699475 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.699456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.703026 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.702993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4" event={"ID":"14fec963-1510-4b5b-be31-706b04c39634","Type":"ContainerStarted","Data":"964beec66770cd95bfc2d8ed0aba92675283fe638f0547a78a43b8988dd8838b"}
Jul 16 21:51:11.713707 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.713641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdd7\" (UniqueName: \"kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7\") pod \"console-65d88997bc-2t2bg\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") " pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:11.883659 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:11.883566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:18.475137 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:18.475105    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:51:20.623898 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:20.623750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fcb87f568-6gz4t"
Jul 16 21:51:23.499440 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.499297    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerName="registry" containerID="cri-o://c68c2a807fac682beeb8ed0233d77505b7c23e2c6236e61f02138e92fb0064b5" gracePeriod=30
Jul 16 21:51:23.581773 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.581742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:51:23.693652 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:23.693569    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c97f451_8c0c_48a2_a561_23b041585c66.slice/crio-4c48c4d3aa2ad483580622e705d756c545c764e8f58bfa2ed1d5643bae40b62c WatchSource:0}: Error finding container 4c48c4d3aa2ad483580622e705d756c545c764e8f58bfa2ed1d5643bae40b62c: Status 404 returned error can't find the container with id 4c48c4d3aa2ad483580622e705d756c545c764e8f58bfa2ed1d5643bae40b62c
Jul 16 21:51:23.749927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.749892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-z98vj" event={"ID":"bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687","Type":"ContainerStarted","Data":"cd0f58e9cb2c1ab80bd258526560bb400d2aab402e12a34eb1709b6e2d1db3dd"}
Jul 16 21:51:23.751196 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.750124    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:51:23.752075 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.751976    2566 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-z98vj container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" start-of-body=
Jul 16 21:51:23.752075 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.752039    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-z98vj" podUID="bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused"
Jul 16 21:51:23.752501 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.752388    2566 generic.go:356] "Generic (PLEG): container finished" podID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerID="c68c2a807fac682beeb8ed0233d77505b7c23e2c6236e61f02138e92fb0064b5" exitCode=0
Jul 16 21:51:23.752501 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.752481    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" event={"ID":"01fb8c89-4b40-433a-9d73-9baa307731fe","Type":"ContainerDied","Data":"c68c2a807fac682beeb8ed0233d77505b7c23e2c6236e61f02138e92fb0064b5"}
Jul 16 21:51:23.754909 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.754492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c65fb7f45-pckhx" event={"ID":"3f03ee34-31ee-46ed-9bbe-67e69da979ce","Type":"ContainerStarted","Data":"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"}
Jul 16 21:51:23.756429 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.756408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wjrxm" event={"ID":"89297fd4-1e1b-42b3-aaec-182a16f6a4b9","Type":"ContainerStarted","Data":"68dada5ca269b71926a2f63a70d6ead08aa7882d24fa3df43a558b410a271b97"}
Jul 16 21:51:23.757662 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.757623    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d88997bc-2t2bg" event={"ID":"2c97f451-8c0c-48a2-a561-23b041585c66","Type":"ContainerStarted","Data":"4c48c4d3aa2ad483580622e705d756c545c764e8f58bfa2ed1d5643bae40b62c"}
Jul 16 21:51:23.759953 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.759897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p9b47" event={"ID":"7a7e0641-b4ef-4104-8a71-b9795f4c6340","Type":"ContainerStarted","Data":"0f0ece51e46efa66ee6c54af2d92dacdbb5c7f749406bf4936fba7d1b3aa16b3"}
Jul 16 21:51:23.773633 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.773331    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-z98vj" podStartSLOduration=1.844624328 podStartE2EDuration="25.77330854s" totalImagesPullingTime="23.928684212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:23.770484979 +0000 UTC m=+83.865996300" watchObservedRunningTime="2026-07-16 21:51:23.77330854 +0000 UTC m=+83.868819858"
Jul 16 21:51:23.794210 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.794149    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c65fb7f45-pckhx" podStartSLOduration=1.453892598 podStartE2EDuration="16.794128464s" totalImagesPullingTime="15.340235866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:23.792521045 +0000 UTC m=+83.888032364" watchObservedRunningTime="2026-07-16 21:51:23.794128464 +0000 UTC m=+83.889639773"
Jul 16 21:51:23.812096 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.812035    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wjrxm" podStartSLOduration=67.286870022 podStartE2EDuration="1m22.812013653s" totalImagesPullingTime="15.525143631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:23.810366579 +0000 UTC m=+83.905877899" watchObservedRunningTime="2026-07-16 21:51:23.812013653 +0000 UTC m=+83.907524975"
Jul 16 21:51:23.832434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.832377    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p9b47" podStartSLOduration=17.210869883 podStartE2EDuration="46.832357514s" totalImagesPullingTime="29.621487631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:23.83133056 +0000 UTC m=+83.926841875" watchObservedRunningTime="2026-07-16 21:51:23.832357514 +0000 UTC m=+83.927868822"
Jul 16 21:51:23.840456 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.840376    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:51:23.900953 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.900921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.900997    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2bnc\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901032    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901108    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.901187 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates\") pod \"01fb8c89-4b40-433a-9d73-9baa307731fe\" (UID: \"01fb8c89-4b40-433a-9d73-9baa307731fe\") "
Jul 16 21:51:23.902626 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.901919    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.904387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.903803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.905194 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.905148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc" (OuterVolumeSpecName: "kube-api-access-g2bnc") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "kube-api-access-g2bnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:23.906105 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.906058    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:23.906528 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.906503    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:23.907396 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.907352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:23.908263 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.908233    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:23.915496 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:23.915461    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "01fb8c89-4b40-433a-9d73-9baa307731fe" (UID: "01fb8c89-4b40-433a-9d73-9baa307731fe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:51:24.002808 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.002773    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-trusted-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003064 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003047    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2bnc\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-kube-api-access-g2bnc\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003166 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003156    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-tls\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003242 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003229    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-image-registry-private-configuration\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003305 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003295    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fb8c89-4b40-433a-9d73-9baa307731fe-installation-pull-secrets\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003368 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003360    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fb8c89-4b40-433a-9d73-9baa307731fe-bound-sa-token\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003424 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003414    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fb8c89-4b40-433a-9d73-9baa307731fe-ca-trust-extracted\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.003490 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.003482    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fb8c89-4b40-433a-9d73-9baa307731fe-registry-certificates\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.766045 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.765999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4" event={"ID":"14fec963-1510-4b5b-be31-706b04c39634","Type":"ContainerStarted","Data":"e61c976d49846faf10e4a9405f16ce4229c11d07e190bc8d86d1021f907c86c7"}
Jul 16 21:51:24.766530 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.766191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:24.768159 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.768127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d88997bc-2t2bg" event={"ID":"2c97f451-8c0c-48a2-a561-23b041585c66","Type":"ContainerStarted","Data":"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"}
Jul 16 21:51:24.770230 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.770007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-645c97ddf8-hctht" event={"ID":"01fb8c89-4b40-433a-9d73-9baa307731fe","Type":"ContainerDied","Data":"aad9602eb52d6537115ef318289b6dd923e08c0c564e0001337ddf097cda8c29"}
Jul 16 21:51:24.770230 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.770096    2566 scope.go:117] "RemoveContainer" containerID="c68c2a807fac682beeb8ed0233d77505b7c23e2c6236e61f02138e92fb0064b5"
Jul 16 21:51:24.770230 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.770018    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-645c97ddf8-hctht"
Jul 16 21:51:24.772860 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.772840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4"
Jul 16 21:51:24.781801 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.781779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-z98vj"
Jul 16 21:51:24.787467 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.787422    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8fgw4" podStartSLOduration=3.930798555 podStartE2EDuration="16.787402079s" totalImagesPullingTime="12.856603524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:24.785565333 +0000 UTC m=+84.881076652" watchObservedRunningTime="2026-07-16 21:51:24.787402079 +0000 UTC m=+84.882913397"
Jul 16 21:51:24.805856 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.805794    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:51:24.809550 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:24.809518    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-645c97ddf8-hctht"]
Jul 16 21:51:25.910126 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.910063    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65d88997bc-2t2bg" podStartSLOduration=14.910040307 podStartE2EDuration="14.910040307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:24.886460535 +0000 UTC m=+84.981971853" watchObservedRunningTime="2026-07-16 21:51:25.910040307 +0000 UTC m=+86.005551626"
Jul 16 21:51:25.911754 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.911721    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"]
Jul 16 21:51:25.912122 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.912101    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerName="registry"
Jul 16 21:51:25.912122 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.912120    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerName="registry"
Jul 16 21:51:25.912303 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.912182    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" containerName="registry"
Jul 16 21:51:25.978479 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.978442    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"]
Jul 16 21:51:25.978727 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.978558    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:25.990326 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.990302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 21:51:25.991434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.991405    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 21:51:25.991434 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.991431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:51:25.991597 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.991447    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2gk7r\""
Jul 16 21:51:25.991597 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.991472    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:51:25.991597 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:25.991477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:26.122383 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.122330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c605f47-1e34-4980-aeb7-f3999b6492d9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.122383 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.122396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.122619 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.122524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hpn\" (UniqueName: \"kubernetes.io/projected/8c605f47-1e34-4980-aeb7-f3999b6492d9-kube-api-access-c7hpn\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.122619 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.122596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.223391 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.223354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.223575 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.223421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c605f47-1e34-4980-aeb7-f3999b6492d9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.223575 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.223454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.223575 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.223525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hpn\" (UniqueName: \"kubernetes.io/projected/8c605f47-1e34-4980-aeb7-f3999b6492d9-kube-api-access-c7hpn\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.223575 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:26.223529    2566 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 21:51:26.223792 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:26.223629    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls podName:8c605f47-1e34-4980-aeb7-f3999b6492d9 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:26.723607603 +0000 UTC m=+86.819118900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-x9pg7" (UID: "8c605f47-1e34-4980-aeb7-f3999b6492d9") : secret "prometheus-operator-tls" not found
Jul 16 21:51:26.224217 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.224191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c605f47-1e34-4980-aeb7-f3999b6492d9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.226449 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.226424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.244861 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.244810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hpn\" (UniqueName: \"kubernetes.io/projected/8c605f47-1e34-4980-aeb7-f3999b6492d9-kube-api-access-c7hpn\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.319447 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.319410    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fb8c89-4b40-433a-9d73-9baa307731fe" path="/var/lib/kubelet/pods/01fb8c89-4b40-433a-9d73-9baa307731fe/volumes"
Jul 16 21:51:26.727725 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.727679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.730582 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.730551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c605f47-1e34-4980-aeb7-f3999b6492d9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-x9pg7\" (UID: \"8c605f47-1e34-4980-aeb7-f3999b6492d9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:26.889480 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:26.889446    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"
Jul 16 21:51:27.089386 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:27.089342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7"]
Jul 16 21:51:27.093698 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:27.093669    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c605f47_1e34_4980_aeb7_f3999b6492d9.slice/crio-357e1b3aab1b1b1099e078de0c78bfc30eeaf20562688448c8f06d94536ae216 WatchSource:0}: Error finding container 357e1b3aab1b1b1099e078de0c78bfc30eeaf20562688448c8f06d94536ae216: Status 404 returned error can't find the container with id 357e1b3aab1b1b1099e078de0c78bfc30eeaf20562688448c8f06d94536ae216
Jul 16 21:51:27.785212 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:27.785165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7" event={"ID":"8c605f47-1e34-4980-aeb7-f3999b6492d9","Type":"ContainerStarted","Data":"357e1b3aab1b1b1099e078de0c78bfc30eeaf20562688448c8f06d94536ae216"}
Jul 16 21:51:27.912554 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:27.911729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:27.912554 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:27.911869    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:27.918643 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:27.918393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:28.795066 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:28.795038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:29.796524 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:29.796480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7" event={"ID":"8c605f47-1e34-4980-aeb7-f3999b6492d9","Type":"ContainerStarted","Data":"0d60c102dd90163e55c1be32856c2bb3a4fd083078231dbf217f7e51e7314360"}
Jul 16 21:51:29.797036 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:29.796586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7" event={"ID":"8c605f47-1e34-4980-aeb7-f3999b6492d9","Type":"ContainerStarted","Data":"89506e403e30f6d48df5cbc39b69db69122722eab7cc5530e99f76eea2f7ee22"}
Jul 16 21:51:29.819092 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:29.819037    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-x9pg7" podStartSLOduration=2.700877535 podStartE2EDuration="4.819016826s" totalImagesPullingTime="2.118139291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:29.817928334 +0000 UTC m=+89.913439653" watchObservedRunningTime="2026-07-16 21:51:29.819016826 +0000 UTC m=+89.914528145"
Jul 16 21:51:31.422655 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.422618    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-d4stg"]
Jul 16 21:51:31.477649 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.477610    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-d4stg"]
Jul 16 21:51:31.477833 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.477679    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hsx9k"]
Jul 16 21:51:31.477903 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.477867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.483929 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.483903    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 21:51:31.484073 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.483960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:51:31.484073 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.483998    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f6cdk\""
Jul 16 21:51:31.484252 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.484238    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 21:51:31.502962 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.502935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.508648 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.508622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l45wb\""
Jul 16 21:51:31.508796 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.508622    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:31.508966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.508946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:31.509261 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.509242    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:31.570695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-tls\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/51dceddc-d757-42c7-aecc-15ca5648ff0c-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4nq\" (UniqueName: \"kubernetes.io/projected/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-api-access-bp4nq\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570888    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.570927 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfr4v\" (UniqueName: \"kubernetes.io/projected/30f239df-f578-4e0e-90a0-eebbdcd81636-kube-api-access-sfr4v\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.570989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-textfile\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-wtmp\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-metrics-client-ca\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.571198 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-root\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.571387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.571387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.571272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-sys\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.672383 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-root\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672429    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-sys\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-tls\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-root\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/51dceddc-d757-42c7-aecc-15ca5648ff0c-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.672579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-sys\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4nq\" (UniqueName: \"kubernetes.io/projected/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-api-access-bp4nq\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfr4v\" (UniqueName: \"kubernetes.io/projected/30f239df-f578-4e0e-90a0-eebbdcd81636-kube-api-access-sfr4v\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-textfile\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-wtmp\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.672893    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-metrics-client-ca\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673048 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.673013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/51dceddc-d757-42c7-aecc-15ca5648ff0c-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.673532 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.673292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-textfile\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673532 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.673495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.673532 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.673504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-metrics-client-ca\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673668 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:31.673600    2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 16 21:51:31.673668 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.673648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-wtmp\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.673668 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:31.673657    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls podName:51dceddc-d757-42c7-aecc-15ca5648ff0c nodeName:}" failed. No retries permitted until 2026-07-16 21:51:32.17363948 +0000 UTC m=+92.269150779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-d4stg" (UID: "51dceddc-d757-42c7-aecc-15ca5648ff0c") : secret "kube-state-metrics-tls" not found
Jul 16 21:51:31.676213 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.676183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.679176 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.677640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-accelerators-collector-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.683535 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.683512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-tls\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.683670 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.683535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/30f239df-f578-4e0e-90a0-eebbdcd81636-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.683751 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.683654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.689864 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.689841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfr4v\" (UniqueName: \"kubernetes.io/projected/30f239df-f578-4e0e-90a0-eebbdcd81636-kube-api-access-sfr4v\") pod \"node-exporter-hsx9k\" (UID: \"30f239df-f578-4e0e-90a0-eebbdcd81636\") " pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.691137 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.691115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4nq\" (UniqueName: \"kubernetes.io/projected/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-api-access-bp4nq\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:31.815123 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.815089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hsx9k"
Jul 16 21:51:31.855895 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:31.855854    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f239df_f578_4e0e_90a0_eebbdcd81636.slice/crio-e3222b56d91e11fda89687110c5a9e1562a3459dad43d28317c58694ec76056f WatchSource:0}: Error finding container e3222b56d91e11fda89687110c5a9e1562a3459dad43d28317c58694ec76056f: Status 404 returned error can't find the container with id e3222b56d91e11fda89687110c5a9e1562a3459dad43d28317c58694ec76056f
Jul 16 21:51:31.884693 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.884648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:31.884890 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.884756    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:31.890265 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:31.890242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:32.177206 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.177159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:32.180163 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.180135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/51dceddc-d757-42c7-aecc-15ca5648ff0c-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-d4stg\" (UID: \"51dceddc-d757-42c7-aecc-15ca5648ff0c\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:32.389151 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.389114    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg"
Jul 16 21:51:32.594339 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.594308    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-d4stg"]
Jul 16 21:51:32.596538 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:32.596510    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dceddc_d757_42c7_aecc_15ca5648ff0c.slice/crio-c3e28d024a474b009e34c05970c4868cc5d6b970541fee6ca1ab7365be44bdbb WatchSource:0}: Error finding container c3e28d024a474b009e34c05970c4868cc5d6b970541fee6ca1ab7365be44bdbb: Status 404 returned error can't find the container with id c3e28d024a474b009e34c05970c4868cc5d6b970541fee6ca1ab7365be44bdbb
Jul 16 21:51:32.814998 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.814899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsx9k" event={"ID":"30f239df-f578-4e0e-90a0-eebbdcd81636","Type":"ContainerStarted","Data":"e3222b56d91e11fda89687110c5a9e1562a3459dad43d28317c58694ec76056f"}
Jul 16 21:51:32.817731 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.817673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg" event={"ID":"51dceddc-d757-42c7-aecc-15ca5648ff0c","Type":"ContainerStarted","Data":"c3e28d024a474b009e34c05970c4868cc5d6b970541fee6ca1ab7365be44bdbb"}
Jul 16 21:51:32.823445 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.823420    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:51:32.877402 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:32.877371    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:51:33.436108 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.436077    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-59b9bd979b-kffjz"]
Jul 16 21:51:33.451943 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.451919    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.462426 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.462388    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 21:51:33.462839 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.462756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k68z2\""
Jul 16 21:51:33.462839 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.462756    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 21:51:33.463701 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.463678    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8lmg1775c3v9d\""
Jul 16 21:51:33.463796 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.463745    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 21:51:33.463876 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.463748    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 21:51:33.463955 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.463939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 21:51:33.465720 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.465700    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b9bd979b-kffjz"]
Jul 16 21:51:33.592742 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.592924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9dc45a7-7a38-466e-befc-08d99086674a-metrics-client-ca\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.592924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592782    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.592924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-grpc-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.592924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.592924 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.593082 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.593082 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.592990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4ht\" (UniqueName: \"kubernetes.io/projected/f9dc45a7-7a38-466e-befc-08d99086674a-kube-api-access-gw4ht\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694287 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4ht\" (UniqueName: \"kubernetes.io/projected/f9dc45a7-7a38-466e-befc-08d99086674a-kube-api-access-gw4ht\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9dc45a7-7a38-466e-befc-08d99086674a-metrics-client-ca\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-grpc-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.694691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.694555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.695805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.695743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9dc45a7-7a38-466e-befc-08d99086674a-metrics-client-ca\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.698030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.697974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.698138 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.698029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.698138 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.698064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.698671 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.698650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.699776 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.699751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-grpc-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.700003 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.699984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f9dc45a7-7a38-466e-befc-08d99086674a-secret-thanos-querier-tls\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.706176 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.706152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4ht\" (UniqueName: \"kubernetes.io/projected/f9dc45a7-7a38-466e-befc-08d99086674a-kube-api-access-gw4ht\") pod \"thanos-querier-59b9bd979b-kffjz\" (UID: \"f9dc45a7-7a38-466e-befc-08d99086674a\") " pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.761105 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.761068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:33.824904 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.824864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsx9k" event={"ID":"30f239df-f578-4e0e-90a0-eebbdcd81636","Type":"ContainerStarted","Data":"8cb40ddf12d9e84fe720df7628870f8640d09eb8e01636061ae6d605e2935b38"}
Jul 16 21:51:33.946325 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:33.946296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-59b9bd979b-kffjz"]
Jul 16 21:51:33.979919 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:33.979890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9dc45a7_7a38_466e_befc_08d99086674a.slice/crio-d64df2685d3add1149e07db53518575f53a2c02e35a4d4abd29db1704ca27320 WatchSource:0}: Error finding container d64df2685d3add1149e07db53518575f53a2c02e35a4d4abd29db1704ca27320: Status 404 returned error can't find the container with id d64df2685d3add1149e07db53518575f53a2c02e35a4d4abd29db1704ca27320
Jul 16 21:51:34.831974 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:34.831941    2566 generic.go:356] "Generic (PLEG): container finished" podID="30f239df-f578-4e0e-90a0-eebbdcd81636" containerID="8cb40ddf12d9e84fe720df7628870f8640d09eb8e01636061ae6d605e2935b38" exitCode=0
Jul 16 21:51:34.832379 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:34.832032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsx9k" event={"ID":"30f239df-f578-4e0e-90a0-eebbdcd81636","Type":"ContainerDied","Data":"8cb40ddf12d9e84fe720df7628870f8640d09eb8e01636061ae6d605e2935b38"}
Jul 16 21:51:34.833218 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:34.833194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"d64df2685d3add1149e07db53518575f53a2c02e35a4d4abd29db1704ca27320"}
Jul 16 21:51:35.839696 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.839657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg" event={"ID":"51dceddc-d757-42c7-aecc-15ca5648ff0c","Type":"ContainerStarted","Data":"33059a87570c58a3a3cd0bf6a95e130a807c272b875d30611f3765136b54610f"}
Jul 16 21:51:35.840182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.839736    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg" event={"ID":"51dceddc-d757-42c7-aecc-15ca5648ff0c","Type":"ContainerStarted","Data":"62cac8c96cef717f93129be5ce153235d4ce91eac0c63f65c02ea45b5b499fb0"}
Jul 16 21:51:35.840182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.839769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg" event={"ID":"51dceddc-d757-42c7-aecc-15ca5648ff0c","Type":"ContainerStarted","Data":"aeafdf16f5f2ba1399427187ec109d7f0cf901e1d2bcf30e00e710abaa12ffff"}
Jul 16 21:51:35.841996 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.841969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsx9k" event={"ID":"30f239df-f578-4e0e-90a0-eebbdcd81636","Type":"ContainerStarted","Data":"104c263563212dbcc364ad8e6952e84b8d6a50d0138082e9adca07f4694acab6"}
Jul 16 21:51:35.842124 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.842016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hsx9k" event={"ID":"30f239df-f578-4e0e-90a0-eebbdcd81636","Type":"ContainerStarted","Data":"ba319fc0d6ecedb1f2e54e9a497112c22e8d5f482143eceee3f8c663e28f15e6"}
Jul 16 21:51:35.898642 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:35.898585    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-d4stg" podStartSLOduration=2.512530189 podStartE2EDuration="4.898562669s" totalImagesPullingTime="2.38603248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:35.896588127 +0000 UTC m=+95.992099445" watchObservedRunningTime="2026-07-16 21:51:35.898562669 +0000 UTC m=+95.994073985"
Jul 16 21:51:36.126355 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.126246    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hsx9k" podStartSLOduration=3.219213856 podStartE2EDuration="5.126222601s" totalImagesPullingTime="1.791265808s" totalInitContainerRuntime="115.742937ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:35.943508222 +0000 UTC m=+96.039019539" watchObservedRunningTime="2026-07-16 21:51:36.126222601 +0000 UTC m=+96.221733919"
Jul 16 21:51:36.126519 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.126454    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:51:36.149898 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.149864    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.153833 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.153764    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:51:36.320721 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320721 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.320993 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.320924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d26q8\" (UniqueName: \"kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421617 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421617 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421563    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.421908 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.422138 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.421953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d26q8\" (UniqueName: \"kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.422347 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.422323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.422748 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.422680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.422748 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.422732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.422946 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.422921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.425111 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.425092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.425310 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.425290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.432920 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.432810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d26q8\" (UniqueName: \"kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8\") pod \"console-575c5f7647-sm9bq\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") " pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.462954 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.462916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:36.765318 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.765287    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:51:36.768333 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:36.768298    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b7524f_7897_4963_8f7a_bb3617f04f64.slice/crio-80760bf3d55008b1e52708c0b6e6217290a2ac1118f6dda0f34cefe6813da790 WatchSource:0}: Error finding container 80760bf3d55008b1e52708c0b6e6217290a2ac1118f6dda0f34cefe6813da790: Status 404 returned error can't find the container with id 80760bf3d55008b1e52708c0b6e6217290a2ac1118f6dda0f34cefe6813da790
Jul 16 21:51:36.850578 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:36.850542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5f7647-sm9bq" event={"ID":"95b7524f-7897-4963-8f7a-bb3617f04f64","Type":"ContainerStarted","Data":"80760bf3d55008b1e52708c0b6e6217290a2ac1118f6dda0f34cefe6813da790"}
Jul 16 21:51:37.856206 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:37.856168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5f7647-sm9bq" event={"ID":"95b7524f-7897-4963-8f7a-bb3617f04f64","Type":"ContainerStarted","Data":"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"}
Jul 16 21:51:37.858390 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:37.858368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"e74a6502ffed364c05a8c8a3872fd698944c2395fb1c84fb32c99a793c631bbb"}
Jul 16 21:51:37.858471 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:37.858434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"20ffd66a595fc3278be1fc325b21c54743bfcffed103e0f4b745e7c1fb02083f"}
Jul 16 21:51:37.858471 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:37.858462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"fe0c20f84eed7a50f11a7b153c024db9b8ea53c8749edaef9a02b5a4c99cfb9d"}
Jul 16 21:51:37.887884 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:37.887800    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-575c5f7647-sm9bq" podStartSLOduration=1.887775563 podStartE2EDuration="1.887775563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:37.886078117 +0000 UTC m=+97.981589447" watchObservedRunningTime="2026-07-16 21:51:37.887775563 +0000 UTC m=+97.983286877"
Jul 16 21:51:38.864498 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:38.864466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"751a653da0708964e07914f20ae5f04ef5189d50ffcc1726580a83e286f74451"}
Jul 16 21:51:39.875264 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:39.875180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"0e2073c1a7b58c329df7d65e17fea9606ed4528752cf63c761bfe50b8fc8f02a"}
Jul 16 21:51:39.875264 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:39.875263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" event={"ID":"f9dc45a7-7a38-466e-befc-08d99086674a","Type":"ContainerStarted","Data":"a16e59008829729377075ac7461e88c9771f6ee875ce108ed593c5abeb8e5383"}
Jul 16 21:51:39.875940 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:39.875331    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:40.699944 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:40.699771    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8twsk"
Jul 16 21:51:40.728453 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:40.728396    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz" podStartSLOduration=3.308633969 podStartE2EDuration="7.728370936s" totalImagesPullingTime="4.419736967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:39.909754082 +0000 UTC m=+100.005265399" watchObservedRunningTime="2026-07-16 21:51:40.728370936 +0000 UTC m=+100.823882263"
Jul 16 21:51:41.050632 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.050596    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:51:41.107662 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.107627    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:51:41.130101 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.130067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.131691 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.131668    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:51:41.267559 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267559 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267837 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267837 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267603    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267837 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267837 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.267837 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.267726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9t2\" (UniqueName: \"kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9t2\" (UniqueName: \"kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369314 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369314 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369274    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369314 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369459 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.369884 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.369857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.370168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.370012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.370168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.370044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.370168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.370015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.371857 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.371837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.371857 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.371855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.387418 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.387394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9t2\" (UniqueName: \"kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2\") pod \"console-7b575d878f-98ckk\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") " pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.440398 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.440371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:41.616471 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.616448    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:51:41.618416 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:51:41.618390    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b248a26_347d_4db5_bbe5_51936ed9739a.slice/crio-0f5ec5a27dab3ea4a3b89e3007aab13c275ef0c36bc8afdb80a2144faa615021 WatchSource:0}: Error finding container 0f5ec5a27dab3ea4a3b89e3007aab13c275ef0c36bc8afdb80a2144faa615021: Status 404 returned error can't find the container with id 0f5ec5a27dab3ea4a3b89e3007aab13c275ef0c36bc8afdb80a2144faa615021
Jul 16 21:51:41.885228 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.885127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b575d878f-98ckk" event={"ID":"2b248a26-347d-4db5-bbe5-51936ed9739a","Type":"ContainerStarted","Data":"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"}
Jul 16 21:51:41.885228 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.885218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b575d878f-98ckk" event={"ID":"2b248a26-347d-4db5-bbe5-51936ed9739a","Type":"ContainerStarted","Data":"0f5ec5a27dab3ea4a3b89e3007aab13c275ef0c36bc8afdb80a2144faa615021"}
Jul 16 21:51:41.912159 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:41.912110    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b575d878f-98ckk" podStartSLOduration=0.912093891 podStartE2EDuration="912.093891ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:41.911505394 +0000 UTC m=+102.007016711" watchObservedRunningTime="2026-07-16 21:51:41.912093891 +0000 UTC m=+102.007605208"
Jul 16 21:51:45.886509 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:45.886482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-59b9bd979b-kffjz"
Jul 16 21:51:46.464042 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:46.463997    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:51:51.441374 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:51.441341    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:51.441852 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:51.441430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:51.446044 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:51.446023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:51.928536 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:51.928499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:51:52.092605 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:52.092574    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:51:57.905016 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:57.904970    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c65fb7f45-pckhx" podUID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" containerName="console" containerID="cri-o://cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2" gracePeriod=15
Jul 16 21:51:58.140603 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.140579    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:58.203747 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.203713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.203957 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.203773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.203957 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.203801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.204076 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.203973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8drr\" (UniqueName: \"kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.204076 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.204012    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.204076 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.204061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config\") pod \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\" (UID: \"3f03ee34-31ee-46ed-9bbe-67e69da979ce\") "
Jul 16 21:51:58.204269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.204230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:58.204413 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.204386    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:58.204483 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.204467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config" (OuterVolumeSpecName: "console-config") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:58.206290 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.206266    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:58.206390 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.206342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:58.206390 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.206352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr" (OuterVolumeSpecName: "kube-api-access-j8drr") pod "3f03ee34-31ee-46ed-9bbe-67e69da979ce" (UID: "3f03ee34-31ee-46ed-9bbe-67e69da979ce"). InnerVolumeSpecName "kube-api-access-j8drr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:58.305613 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305575    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8drr\" (UniqueName: \"kubernetes.io/projected/3f03ee34-31ee-46ed-9bbe-67e69da979ce-kube-api-access-j8drr\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.305613 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305608    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-service-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.305613 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305617    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.305864 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305626    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.305864 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305636    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f03ee34-31ee-46ed-9bbe-67e69da979ce-oauth-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.305864 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.305645    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f03ee34-31ee-46ed-9bbe-67e69da979ce-console-oauth-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:51:58.952873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.952805    2566 generic.go:356] "Generic (PLEG): container finished" podID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" containerID="cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2" exitCode=0
Jul 16 21:51:58.952873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.952878    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c65fb7f45-pckhx"
Jul 16 21:51:58.953416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.952892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c65fb7f45-pckhx" event={"ID":"3f03ee34-31ee-46ed-9bbe-67e69da979ce","Type":"ContainerDied","Data":"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"}
Jul 16 21:51:58.953416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.952957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c65fb7f45-pckhx" event={"ID":"3f03ee34-31ee-46ed-9bbe-67e69da979ce","Type":"ContainerDied","Data":"aa90df637585e5e3769881663375abf0ecf9fd068e5dfdb3592d862345083d11"}
Jul 16 21:51:58.953416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.952979    2566 scope.go:117] "RemoveContainer" containerID="cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"
Jul 16 21:51:58.961450 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.961432    2566 scope.go:117] "RemoveContainer" containerID="cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"
Jul 16 21:51:58.961720 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:51:58.961698    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2\": container with ID starting with cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2 not found: ID does not exist" containerID="cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"
Jul 16 21:51:58.961766 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.961727    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2"} err="failed to get container status \"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2\": rpc error: code = NotFound desc = could not find container \"cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2\": container with ID starting with cd7d71aaed0173edd4f3b759f5e9b13a1d64db8488756512b216587b5fce97a2 not found: ID does not exist"
Jul 16 21:51:58.970562 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.970540    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:51:58.974580 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:51:58.974561    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c65fb7f45-pckhx"]
Jul 16 21:52:00.319230 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:00.319199    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" path="/var/lib/kubelet/pods/3f03ee34-31ee-46ed-9bbe-67e69da979ce/volumes"
Jul 16 21:52:06.071966 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.071894    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-575c5f7647-sm9bq" podUID="95b7524f-7897-4963-8f7a-bb3617f04f64" containerName="console" containerID="cri-o://c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437" gracePeriod=15
Jul 16 21:52:06.349051 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.348973    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:52:06.372778 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.372749    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.372972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.372787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d26q8\" (UniqueName: \"kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.372972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.372846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.372972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.372892    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.372972 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.372945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.373301 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.373243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:06.373467 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.373426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config" (OuterVolumeSpecName: "console-config") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:06.373549 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.373448    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca" (OuterVolumeSpecName: "service-ca") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:06.373549 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.373535    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.373657 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.373608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config\") pod \"95b7524f-7897-4963-8f7a-bb3617f04f64\" (UID: \"95b7524f-7897-4963-8f7a-bb3617f04f64\") "
Jul 16 21:52:06.374168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.374110    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-trusted-ca-bundle\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.374168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.374133    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-console-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.374168 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.374148    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-service-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.375598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.375561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:06.376389 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.376323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:06.376502 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.376397    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8" (OuterVolumeSpecName: "kube-api-access-d26q8") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "kube-api-access-d26q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:06.376502 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.376439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "95b7524f-7897-4963-8f7a-bb3617f04f64" (UID: "95b7524f-7897-4963-8f7a-bb3617f04f64"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:06.474833 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.474789    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.474833 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.474840    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95b7524f-7897-4963-8f7a-bb3617f04f64-oauth-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.475038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.474854    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95b7524f-7897-4963-8f7a-bb3617f04f64-console-oauth-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.475038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.474865    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d26q8\" (UniqueName: \"kubernetes.io/projected/95b7524f-7897-4963-8f7a-bb3617f04f64-kube-api-access-d26q8\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:06.985012 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.984979    2566 generic.go:356] "Generic (PLEG): container finished" podID="95b7524f-7897-4963-8f7a-bb3617f04f64" containerID="c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437" exitCode=0
Jul 16 21:52:06.985179 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.985020    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-575c5f7647-sm9bq"
Jul 16 21:52:06.985179 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.985064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5f7647-sm9bq" event={"ID":"95b7524f-7897-4963-8f7a-bb3617f04f64","Type":"ContainerDied","Data":"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"}
Jul 16 21:52:06.985179 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.985118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-575c5f7647-sm9bq" event={"ID":"95b7524f-7897-4963-8f7a-bb3617f04f64","Type":"ContainerDied","Data":"80760bf3d55008b1e52708c0b6e6217290a2ac1118f6dda0f34cefe6813da790"}
Jul 16 21:52:06.985179 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.985141    2566 scope.go:117] "RemoveContainer" containerID="c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"
Jul 16 21:52:06.993452 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.993434    2566 scope.go:117] "RemoveContainer" containerID="c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"
Jul 16 21:52:06.993723 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:52:06.993705    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437\": container with ID starting with c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437 not found: ID does not exist" containerID="c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"
Jul 16 21:52:06.993769 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:06.993733    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437"} err="failed to get container status \"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437\": rpc error: code = NotFound desc = could not find container \"c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437\": container with ID starting with c003155ad77549edab8dbdce4cfc8fe61dbefe3973b36bb30677669a4a172437 not found: ID does not exist"
Jul 16 21:52:07.006835 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:07.006791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:52:07.011143 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:07.011122    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-575c5f7647-sm9bq"]
Jul 16 21:52:08.318553 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:08.318513    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95b7524f-7897-4963-8f7a-bb3617f04f64" path="/var/lib/kubelet/pods/95b7524f-7897-4963-8f7a-bb3617f04f64/volumes"
Jul 16 21:52:17.119596 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.119528    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65d88997bc-2t2bg" podUID="2c97f451-8c0c-48a2-a561-23b041585c66" containerName="console" containerID="cri-o://20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5" gracePeriod=15
Jul 16 21:52:17.388281 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.388215    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:52:17.475647 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.475859 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.475859 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.475859 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.476038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475892    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.476038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475949    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.476038 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.475972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vdd7\" (UniqueName: \"kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7\") pod \"2c97f451-8c0c-48a2-a561-23b041585c66\" (UID: \"2c97f451-8c0c-48a2-a561-23b041585c66\") "
Jul 16 21:52:17.476213 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.476185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:17.476261 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.476209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:17.476261 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.476187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config" (OuterVolumeSpecName: "console-config") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:17.476343 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.476262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca" (OuterVolumeSpecName: "service-ca") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:17.478311 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.478281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7" (OuterVolumeSpecName: "kube-api-access-5vdd7") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "kube-api-access-5vdd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:17.478311 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.478289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:17.478441 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.478341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2c97f451-8c0c-48a2-a561-23b041585c66" (UID: "2c97f451-8c0c-48a2-a561-23b041585c66"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:17.576971 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.576935    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-console-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.576971 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.576964    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-trusted-ca-bundle\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.576971 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.576974    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.577182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.576985    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-oauth-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.577182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.576993    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c97f451-8c0c-48a2-a561-23b041585c66-service-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.577182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.577001    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c97f451-8c0c-48a2-a561-23b041585c66-console-oauth-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:17.577182 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:17.577010    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdd7\" (UniqueName: \"kubernetes.io/projected/2c97f451-8c0c-48a2-a561-23b041585c66-kube-api-access-5vdd7\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:52:18.031764 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.031730    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c97f451-8c0c-48a2-a561-23b041585c66" containerID="20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5" exitCode=0
Jul 16 21:52:18.031920 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.031779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d88997bc-2t2bg"
Jul 16 21:52:18.031920 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.031799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d88997bc-2t2bg" event={"ID":"2c97f451-8c0c-48a2-a561-23b041585c66","Type":"ContainerDied","Data":"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"}
Jul 16 21:52:18.031920 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.031855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d88997bc-2t2bg" event={"ID":"2c97f451-8c0c-48a2-a561-23b041585c66","Type":"ContainerDied","Data":"4c48c4d3aa2ad483580622e705d756c545c764e8f58bfa2ed1d5643bae40b62c"}
Jul 16 21:52:18.031920 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.031876    2566 scope.go:117] "RemoveContainer" containerID="20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"
Jul 16 21:52:18.040145 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.040126    2566 scope.go:117] "RemoveContainer" containerID="20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"
Jul 16 21:52:18.040411 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:52:18.040394    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5\": container with ID starting with 20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5 not found: ID does not exist" containerID="20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"
Jul 16 21:52:18.040448 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.040420    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5"} err="failed to get container status \"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5\": rpc error: code = NotFound desc = could not find container \"20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5\": container with ID starting with 20784de276a0d8fc1da5bf06b0733034a15820e3dd890f7389913323578ab7a5 not found: ID does not exist"
Jul 16 21:52:18.052433 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.052407    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:52:18.056857 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.056210    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65d88997bc-2t2bg"]
Jul 16 21:52:18.319301 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:18.319221    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c97f451-8c0c-48a2-a561-23b041585c66" path="/var/lib/kubelet/pods/2c97f451-8c0c-48a2-a561-23b041585c66/volumes"
Jul 16 21:52:54.013130 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013048    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013366    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95b7524f-7897-4963-8f7a-bb3617f04f64" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013380    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95b7524f-7897-4963-8f7a-bb3617f04f64" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013388    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013393    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013400    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c97f451-8c0c-48a2-a561-23b041585c66" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013405    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c97f451-8c0c-48a2-a561-23b041585c66" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013460    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f03ee34-31ee-46ed-9bbe-67e69da979ce" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013470    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95b7524f-7897-4963-8f7a-bb3617f04f64" containerName="console"
Jul 16 21:52:54.013566 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.013476    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c97f451-8c0c-48a2-a561-23b041585c66" containerName="console"
Jul 16 21:52:54.016145 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.016120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.032266 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.032237    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:52:54.071056 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071223 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071223 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071143    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071223 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071223 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071223 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfl84\" (UniqueName: \"kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.071386 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.071264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.171776 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfl84\" (UniqueName: \"kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.171954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172635 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.172600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172758 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.172669    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172758 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.172708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.172925 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.172906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.174646 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.174620    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.174919 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.174896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.181898 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.181876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfl84\" (UniqueName: \"kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84\") pod \"console-ff5d44c7b-cflp6\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") " pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.325552 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.325464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:52:54.464956 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:54.464929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:52:54.467628 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:52:54.467604    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e45c16_7294_4f95_a7b1_b2d56d917286.slice/crio-3873dd7a30dd31a2955d91b1dc8085235b4928375cc79509480a22d5f2850926 WatchSource:0}: Error finding container 3873dd7a30dd31a2955d91b1dc8085235b4928375cc79509480a22d5f2850926: Status 404 returned error can't find the container with id 3873dd7a30dd31a2955d91b1dc8085235b4928375cc79509480a22d5f2850926
Jul 16 21:52:55.169987 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:55.169945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff5d44c7b-cflp6" event={"ID":"d9e45c16-7294-4f95-a7b1-b2d56d917286","Type":"ContainerStarted","Data":"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"}
Jul 16 21:52:55.170491 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:55.170008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff5d44c7b-cflp6" event={"ID":"d9e45c16-7294-4f95-a7b1-b2d56d917286","Type":"ContainerStarted","Data":"3873dd7a30dd31a2955d91b1dc8085235b4928375cc79509480a22d5f2850926"}
Jul 16 21:52:55.193658 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:52:55.193595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ff5d44c7b-cflp6" podStartSLOduration=2.193576325 podStartE2EDuration="2.193576325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:55.192792881 +0000 UTC m=+175.288304196" watchObservedRunningTime="2026-07-16 21:52:55.193576325 +0000 UTC m=+175.289087641"
Jul 16 21:53:04.326221 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:04.326186    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:53:04.326642 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:04.326243    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:53:04.330795 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:04.330767    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:53:05.213008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:05.212980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:53:05.269997 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:05.269969    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:53:30.294496 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.294427    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b575d878f-98ckk" podUID="2b248a26-347d-4db5-bbe5-51936ed9739a" containerName="console" containerID="cri-o://8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6" gracePeriod=15
Jul 16 21:53:30.550225 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.550157    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:53:30.665269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665269 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665482 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665316    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665482 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665344    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665482 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665386    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665482 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665414    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.665632 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.665486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp9t2\" (UniqueName: \"kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2\") pod \"2b248a26-347d-4db5-bbe5-51936ed9739a\" (UID: \"2b248a26-347d-4db5-bbe5-51936ed9739a\") "
Jul 16 21:53:30.666323 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.666220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:30.666323 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.666230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config" (OuterVolumeSpecName: "console-config") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:30.666323 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.666289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:30.666323 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.666283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:30.672310 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.672279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2" (OuterVolumeSpecName: "kube-api-access-lp9t2") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "kube-api-access-lp9t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:53:30.672310 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.672289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:53:30.672472 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.672336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b248a26-347d-4db5-bbe5-51936ed9739a" (UID: "2b248a26-347d-4db5-bbe5-51936ed9739a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:53:30.767569 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767532    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9t2\" (UniqueName: \"kubernetes.io/projected/2b248a26-347d-4db5-bbe5-51936ed9739a-kube-api-access-lp9t2\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767569 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767564    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767569 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767576    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-console-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767799 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767585    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-oauth-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767799 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767593    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-trusted-ca-bundle\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767799 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767601    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b248a26-347d-4db5-bbe5-51936ed9739a-service-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:30.767799 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:30.767609    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b248a26-347d-4db5-bbe5-51936ed9739a-console-oauth-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:53:31.304512 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.304471    2566 generic.go:356] "Generic (PLEG): container finished" podID="2b248a26-347d-4db5-bbe5-51936ed9739a" containerID="8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6" exitCode=0
Jul 16 21:53:31.304512 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.304514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b575d878f-98ckk" event={"ID":"2b248a26-347d-4db5-bbe5-51936ed9739a","Type":"ContainerDied","Data":"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"}
Jul 16 21:53:31.305078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.304541    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b575d878f-98ckk"
Jul 16 21:53:31.305078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.304582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b575d878f-98ckk" event={"ID":"2b248a26-347d-4db5-bbe5-51936ed9739a","Type":"ContainerDied","Data":"0f5ec5a27dab3ea4a3b89e3007aab13c275ef0c36bc8afdb80a2144faa615021"}
Jul 16 21:53:31.305078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.304603    2566 scope.go:117] "RemoveContainer" containerID="8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"
Jul 16 21:53:31.313142 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.313124    2566 scope.go:117] "RemoveContainer" containerID="8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"
Jul 16 21:53:31.313420 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:53:31.313399    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6\": container with ID starting with 8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6 not found: ID does not exist" containerID="8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"
Jul 16 21:53:31.313497 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.313439    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6"} err="failed to get container status \"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6\": rpc error: code = NotFound desc = could not find container \"8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6\": container with ID starting with 8f96681eb8918683162d6a05e1bb0eb06709f6a1b6be536462b19f6b3fd862d6 not found: ID does not exist"
Jul 16 21:53:31.325787 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.325761    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:53:31.331399 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:31.331368    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b575d878f-98ckk"]
Jul 16 21:53:32.318272 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:53:32.318241    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b248a26-347d-4db5-bbe5-51936ed9739a" path="/var/lib/kubelet/pods/2b248a26-347d-4db5-bbe5-51936ed9739a/volumes"
Jul 16 21:54:07.874868 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.874831    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"]
Jul 16 21:54:07.875367 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.875120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b248a26-347d-4db5-bbe5-51936ed9739a" containerName="console"
Jul 16 21:54:07.875367 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.875131    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b248a26-347d-4db5-bbe5-51936ed9739a" containerName="console"
Jul 16 21:54:07.875367 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.875192    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b248a26-347d-4db5-bbe5-51936ed9739a" containerName="console"
Jul 16 21:54:07.879301 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.879281    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:07.882254 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.882228    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 21:54:07.882375 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.882258    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-xvsxx\""
Jul 16 21:54:07.882375 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.882297    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 21:54:07.882375 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.882258    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 21:54:07.892166 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.892137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"]
Jul 16 21:54:07.951990 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.951953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ff614731-0bbc-44ec-a447-5f83669f4a09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:07.952152 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:07.952009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98pr\" (UniqueName: \"kubernetes.io/projected/ff614731-0bbc-44ec-a447-5f83669f4a09-kube-api-access-c98pr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.052765 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.052720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ff614731-0bbc-44ec-a447-5f83669f4a09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.052964 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.052779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98pr\" (UniqueName: \"kubernetes.io/projected/ff614731-0bbc-44ec-a447-5f83669f4a09-kube-api-access-c98pr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.055237 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.055214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ff614731-0bbc-44ec-a447-5f83669f4a09-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.062394 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.062370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98pr\" (UniqueName: \"kubernetes.io/projected/ff614731-0bbc-44ec-a447-5f83669f4a09-kube-api-access-c98pr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-26g9l\" (UID: \"ff614731-0bbc-44ec-a447-5f83669f4a09\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.193745 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.193646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:08.354109 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.354084    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"]
Jul 16 21:54:08.357042 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:54:08.357010    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff614731_0bbc_44ec_a447_5f83669f4a09.slice/crio-6f2ad7e92009a8a8da9b95df30f080767292d9942a82b7f43b0fdc06fc78572d WatchSource:0}: Error finding container 6f2ad7e92009a8a8da9b95df30f080767292d9942a82b7f43b0fdc06fc78572d: Status 404 returned error can't find the container with id 6f2ad7e92009a8a8da9b95df30f080767292d9942a82b7f43b0fdc06fc78572d
Jul 16 21:54:08.445285 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:08.445197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l" event={"ID":"ff614731-0bbc-44ec-a447-5f83669f4a09","Type":"ContainerStarted","Data":"6f2ad7e92009a8a8da9b95df30f080767292d9942a82b7f43b0fdc06fc78572d"}
Jul 16 21:54:12.465686 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:12.465652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l" event={"ID":"ff614731-0bbc-44ec-a447-5f83669f4a09","Type":"ContainerStarted","Data":"fa960ea06c9b7186b16be482fbeeb986fc7abb1942db0a50aa284a467e9aa879"}
Jul 16 21:54:12.466099 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:12.465772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:54:12.495347 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:12.495290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l" podStartSLOduration=2.217046626 podStartE2EDuration="5.495277549s" totalImagesPullingTime="3.278230923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:12.494242132 +0000 UTC m=+252.589753460" watchObservedRunningTime="2026-07-16 21:54:12.495277549 +0000 UTC m=+252.590788878"
Jul 16 21:54:33.472907 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:54:33.472803    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-26g9l"
Jul 16 21:55:00.222065 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:00.222032    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:55:00.228544 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:00.228508    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:55:00.229790 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:00.229768    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 21:55:21.202028 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.201993    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 21:55:21.205426 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.205403    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.207780 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.207761    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 16 21:55:21.207982 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.207963    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-vmwls\""
Jul 16 21:55:21.209012 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.208992    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 21:55:21.209127 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.209040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 21:55:21.215558 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.215533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 21:55:21.229104 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.229078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.229197 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.229125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp78n\" (UniqueName: \"kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.329797 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.329762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.329979 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.329844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp78n\" (UniqueName: \"kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.329979 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:55:21.329932    2566 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 16 21:55:21.330056 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:55:21.330000    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert podName:3cd417e6-878a-4e13-bef1-cafc23df6df6 nodeName:}" failed. No retries permitted until 2026-07-16 21:55:21.829984051 +0000 UTC m=+321.925495346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert") pod "llmisvc-controller-manager-78594f6654-2bbvp" (UID: "3cd417e6-878a-4e13-bef1-cafc23df6df6") : secret "llmisvc-webhook-server-cert" not found
Jul 16 21:55:21.340897 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.340869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp78n\" (UniqueName: \"kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.834257 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.834210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:21.836721 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:21.836691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") pod \"llmisvc-controller-manager-78594f6654-2bbvp\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") " pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:22.117093 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:22.116997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:22.245983 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:22.245956    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 21:55:22.248250 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:55:22.248216    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3cd417e6_878a_4e13_bef1_cafc23df6df6.slice/crio-56cfb2931995f322ee9f8703b77bf935cf55c772c7dc7655ae56e0a3365e68db WatchSource:0}: Error finding container 56cfb2931995f322ee9f8703b77bf935cf55c772c7dc7655ae56e0a3365e68db: Status 404 returned error can't find the container with id 56cfb2931995f322ee9f8703b77bf935cf55c772c7dc7655ae56e0a3365e68db
Jul 16 21:55:22.249648 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:22.249625    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:55:22.735638 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:22.735602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" event={"ID":"3cd417e6-878a-4e13-bef1-cafc23df6df6","Type":"ContainerStarted","Data":"56cfb2931995f322ee9f8703b77bf935cf55c772c7dc7655ae56e0a3365e68db"}
Jul 16 21:55:25.753123 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:25.753087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" event={"ID":"3cd417e6-878a-4e13-bef1-cafc23df6df6","Type":"ContainerStarted","Data":"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"}
Jul 16 21:55:25.753529 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:25.753305    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:55:25.772628 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:25.772575    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" podStartSLOduration=1.692547495 podStartE2EDuration="4.772560148s" totalImagesPullingTime="3.080012653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:25.770856931 +0000 UTC m=+325.866368240" watchObservedRunningTime="2026-07-16 21:55:25.772560148 +0000 UTC m=+325.868071465"
Jul 16 21:55:56.760077 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:55:56.760004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 21:56:18.052638 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.052599    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"]
Jul 16 21:56:18.059910 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.059893    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.063294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.063263    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 16 21:56:18.063294 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.063293    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:56:18.063460 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.063316    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r56hg\""
Jul 16 21:56:18.072781 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.072759    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"]
Jul 16 21:56:18.175206 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.175168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f62ea-69e6-4dfb-9d87-5defcd75afbc-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.175206 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.175211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn6v\" (UniqueName: \"kubernetes.io/projected/911f62ea-69e6-4dfb-9d87-5defcd75afbc-kube-api-access-wxn6v\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.276388 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.276340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f62ea-69e6-4dfb-9d87-5defcd75afbc-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.276557 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.276392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn6v\" (UniqueName: \"kubernetes.io/projected/911f62ea-69e6-4dfb-9d87-5defcd75afbc-kube-api-access-wxn6v\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.276738 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.276720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/911f62ea-69e6-4dfb-9d87-5defcd75afbc-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.285683 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.285647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn6v\" (UniqueName: \"kubernetes.io/projected/911f62ea-69e6-4dfb-9d87-5defcd75afbc-kube-api-access-wxn6v\") pod \"cert-manager-operator-controller-manager-684f79c9b9-tfzvx\" (UID: \"911f62ea-69e6-4dfb-9d87-5defcd75afbc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.368997 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.368910    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"
Jul 16 21:56:18.527717 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.527671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx"]
Jul 16 21:56:18.533472 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:56:18.533441    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911f62ea_69e6_4dfb_9d87_5defcd75afbc.slice/crio-fa0eaee73ae8c40f25b7bf0464c41ef219ee49e91eb84a63e56b9dfbbd4d365b WatchSource:0}: Error finding container fa0eaee73ae8c40f25b7bf0464c41ef219ee49e91eb84a63e56b9dfbbd4d365b: Status 404 returned error can't find the container with id fa0eaee73ae8c40f25b7bf0464c41ef219ee49e91eb84a63e56b9dfbbd4d365b
Jul 16 21:56:18.972007 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:18.971964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx" event={"ID":"911f62ea-69e6-4dfb-9d87-5defcd75afbc","Type":"ContainerStarted","Data":"fa0eaee73ae8c40f25b7bf0464c41ef219ee49e91eb84a63e56b9dfbbd4d365b"}
Jul 16 21:56:21.988414 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:21.988370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx" event={"ID":"911f62ea-69e6-4dfb-9d87-5defcd75afbc","Type":"ContainerStarted","Data":"239903a8e8413e7f4deac4e34e372b7cf2b7cee81fa3c51ec6a600322b3aa609"}
Jul 16 21:56:22.035163 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.035108    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-tfzvx" podStartSLOduration=1.275214067 podStartE2EDuration="4.035091387s" totalImagesPullingTime="2.75987732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:22.032909165 +0000 UTC m=+382.128420476" watchObservedRunningTime="2026-07-16 21:56:22.035091387 +0000 UTC m=+382.130602703"
Jul 16 21:56:22.600378 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.600342    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vqsq5"]
Jul 16 21:56:22.603706 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.603688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.612391 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.612375    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 21:56:22.612521 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.612504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zkpvb\""
Jul 16 21:56:22.613608 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.613592    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 21:56:22.627579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.627556    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vqsq5"]
Jul 16 21:56:22.713494 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.713452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.713710 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.713513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qf7\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-kube-api-access-m5qf7\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.814242 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.814202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qf7\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-kube-api-access-m5qf7\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.814416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.814282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.838493 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.838457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.858082 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.858010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qf7\" (UniqueName: \"kubernetes.io/projected/f4f0804b-e244-4789-9c85-375300cc9d45-kube-api-access-m5qf7\") pod \"cert-manager-webhook-6486ffd997-vqsq5\" (UID: \"f4f0804b-e244-4789-9c85-375300cc9d45\") " pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:22.920919 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:22.920884    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:23.108973 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:23.108896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-vqsq5"]
Jul 16 21:56:23.116502 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:56:23.116461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4f0804b_e244_4789_9c85_375300cc9d45.slice/crio-ee20085d8f0f39264f739c1d0f47f1ff10dfcac84b53e8f3e46052962443960b WatchSource:0}: Error finding container ee20085d8f0f39264f739c1d0f47f1ff10dfcac84b53e8f3e46052962443960b: Status 404 returned error can't find the container with id ee20085d8f0f39264f739c1d0f47f1ff10dfcac84b53e8f3e46052962443960b
Jul 16 21:56:24.000593 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.000553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5" event={"ID":"f4f0804b-e244-4789-9c85-375300cc9d45","Type":"ContainerStarted","Data":"ee20085d8f0f39264f739c1d0f47f1ff10dfcac84b53e8f3e46052962443960b"}
Jul 16 21:56:24.690720 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.690685    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"]
Jul 16 21:56:24.694160 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.694137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.700241 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.700209    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nh98g\""
Jul 16 21:56:24.706841 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.706800    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"]
Jul 16 21:56:24.831261 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.831218    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.831436 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.831274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbprl\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-kube-api-access-dbprl\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.932016 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.931973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.932016 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.932012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbprl\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-kube-api-access-dbprl\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.945283 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.945184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:24.953272 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:24.953247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbprl\" (UniqueName: \"kubernetes.io/projected/2757c14f-6e49-404f-a3b5-f9efddbfc8bb-kube-api-access-dbprl\") pod \"cert-manager-cainjector-559cdd8c87-wglmf\" (UID: \"2757c14f-6e49-404f-a3b5-f9efddbfc8bb\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:25.006094 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:25.006061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"
Jul 16 21:56:25.831146 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:25.831115    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wglmf"]
Jul 16 21:56:25.834405 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:56:25.834381    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2757c14f_6e49_404f_a3b5_f9efddbfc8bb.slice/crio-738548a0f229d981d880ca78815d5f4b9b9fb7f2bf7106f94a48993c27698a1a WatchSource:0}: Error finding container 738548a0f229d981d880ca78815d5f4b9b9fb7f2bf7106f94a48993c27698a1a: Status 404 returned error can't find the container with id 738548a0f229d981d880ca78815d5f4b9b9fb7f2bf7106f94a48993c27698a1a
Jul 16 21:56:26.011301 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:26.011267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf" event={"ID":"2757c14f-6e49-404f-a3b5-f9efddbfc8bb","Type":"ContainerStarted","Data":"752689ada0055a91f6f8623464ceb5ed355400c1766d970c8eb22a802a06fe7b"}
Jul 16 21:56:26.011494 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:26.011313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf" event={"ID":"2757c14f-6e49-404f-a3b5-f9efddbfc8bb","Type":"ContainerStarted","Data":"738548a0f229d981d880ca78815d5f4b9b9fb7f2bf7106f94a48993c27698a1a"}
Jul 16 21:56:26.012713 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:26.012680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5" event={"ID":"f4f0804b-e244-4789-9c85-375300cc9d45","Type":"ContainerStarted","Data":"f36b8bf1feb85a72fc8b6e5248ae6bb179133ddd1925c4c58f799f9063b6c212"}
Jul 16 21:56:26.012873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:26.012809    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:26.034351 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:26.034301    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wglmf" podStartSLOduration=2.03428395 podStartE2EDuration="2.03428395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:26.033112273 +0000 UTC m=+386.128623590" watchObservedRunningTime="2026-07-16 21:56:26.03428395 +0000 UTC m=+386.129795266"
Jul 16 21:56:32.019386 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:32.019353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5"
Jul 16 21:56:32.038272 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:32.038220    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-vqsq5" podStartSLOduration=7.415908254 podStartE2EDuration="10.038202887s" totalImagesPullingTime="2.622294633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:26.061071377 +0000 UTC m=+386.156582690" watchObservedRunningTime="2026-07-16 21:56:32.038202887 +0000 UTC m=+392.133714205"
Jul 16 21:56:42.904315 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.904280    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nnhmd"]
Jul 16 21:56:42.907652 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.907636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:42.910555 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.910528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-h6zfs\""
Jul 16 21:56:42.921583 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.921560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nnhmd"]
Jul 16 21:56:42.982213 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.982180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-bound-sa-token\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:42.982378 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:42.982225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vx7\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-kube-api-access-c4vx7\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.083278 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.083246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vx7\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-kube-api-access-c4vx7\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.083438 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.083306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-bound-sa-token\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.092511 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.092483    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-bound-sa-token\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.092647 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.092629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vx7\" (UniqueName: \"kubernetes.io/projected/ce6ab366-054f-4568-ba82-3e9639f4c087-kube-api-access-c4vx7\") pod \"cert-manager-55b4d954d5-nnhmd\" (UID: \"ce6ab366-054f-4568-ba82-3e9639f4c087\") " pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.217030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.216999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nnhmd"
Jul 16 21:56:43.345398 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:43.345365    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nnhmd"]
Jul 16 21:56:43.348151 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:56:43.348123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6ab366_054f_4568_ba82_3e9639f4c087.slice/crio-360dd83a77dece325d6754798c169c008a26b1a91074535f8f27f5fc91bb376d WatchSource:0}: Error finding container 360dd83a77dece325d6754798c169c008a26b1a91074535f8f27f5fc91bb376d: Status 404 returned error can't find the container with id 360dd83a77dece325d6754798c169c008a26b1a91074535f8f27f5fc91bb376d
Jul 16 21:56:44.097663 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:44.097627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nnhmd" event={"ID":"ce6ab366-054f-4568-ba82-3e9639f4c087","Type":"ContainerStarted","Data":"4ecfd434b2af4ecd200d6f7f9a9c695738b22fdbe872ecd9fe152988a1b42fb3"}
Jul 16 21:56:44.098040 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:44.097674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nnhmd" event={"ID":"ce6ab366-054f-4568-ba82-3e9639f4c087","Type":"ContainerStarted","Data":"360dd83a77dece325d6754798c169c008a26b1a91074535f8f27f5fc91bb376d"}
Jul 16 21:56:44.115890 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:44.115839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nnhmd" podStartSLOduration=2.11579686 podStartE2EDuration="2.11579686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:44.115299518 +0000 UTC m=+404.210810836" watchObservedRunningTime="2026-07-16 21:56:44.11579686 +0000 UTC m=+404.211308177"
Jul 16 21:56:50.053663 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.053629    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79c4bccd4b-9kzkn"]
Jul 16 21:56:50.057189 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.057168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.079976 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.079943    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c4bccd4b-9kzkn"]
Jul 16 21:56:50.135387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-console-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135387 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-trusted-ca-bundle\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135556 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135404    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-oauth-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135556 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-service-ca\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135556 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-oauth-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135556 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9kxg\" (UniqueName: \"kubernetes.io/projected/2325992e-3418-4166-9578-9c2dfce4e668-kube-api-access-g9kxg\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.135673 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.135564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236258 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-oauth-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236258 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9kxg\" (UniqueName: \"kubernetes.io/projected/2325992e-3418-4166-9578-9c2dfce4e668-kube-api-access-g9kxg\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-console-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-trusted-ca-bundle\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-oauth-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.236516 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.236388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-service-ca\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.237179 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.237114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-service-ca\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.237296 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.237268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-console-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.237356 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.237325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-trusted-ca-bundle\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.237419 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.237375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2325992e-3418-4166-9578-9c2dfce4e668-oauth-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.238999 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.238979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-oauth-config\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.239097 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.239028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2325992e-3418-4166-9578-9c2dfce4e668-console-serving-cert\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.331046 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.330980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9kxg\" (UniqueName: \"kubernetes.io/projected/2325992e-3418-4166-9578-9c2dfce4e668-kube-api-access-g9kxg\") pod \"console-79c4bccd4b-9kzkn\" (UID: \"2325992e-3418-4166-9578-9c2dfce4e668\") " pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.365902 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.365867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:56:50.512905 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:50.512875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79c4bccd4b-9kzkn"]
Jul 16 21:56:50.516025 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:56:50.516002    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2325992e_3418_4166_9578_9c2dfce4e668.slice/crio-5558f91f43535c1b391134e42a2d9e0b5d4bb51dc041c9fda1b689124d212822 WatchSource:0}: Error finding container 5558f91f43535c1b391134e42a2d9e0b5d4bb51dc041c9fda1b689124d212822: Status 404 returned error can't find the container with id 5558f91f43535c1b391134e42a2d9e0b5d4bb51dc041c9fda1b689124d212822
Jul 16 21:56:51.130230 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:51.130195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c4bccd4b-9kzkn" event={"ID":"2325992e-3418-4166-9578-9c2dfce4e668","Type":"ContainerStarted","Data":"161e2f611104e3535bfdbfe85afeb16c4eb7e9ef98dfa63fa4c3cf63083e3820"}
Jul 16 21:56:51.130654 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:51.130242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79c4bccd4b-9kzkn" event={"ID":"2325992e-3418-4166-9578-9c2dfce4e668","Type":"ContainerStarted","Data":"5558f91f43535c1b391134e42a2d9e0b5d4bb51dc041c9fda1b689124d212822"}
Jul 16 21:56:51.169782 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:56:51.169720    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79c4bccd4b-9kzkn" podStartSLOduration=1.169703652 podStartE2EDuration="1.169703652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:51.169384249 +0000 UTC m=+411.264895690" watchObservedRunningTime="2026-07-16 21:56:51.169703652 +0000 UTC m=+411.265214979"
Jul 16 21:57:00.366473 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:00.366435    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:57:00.366970 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:00.366499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:57:00.371412 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:00.371384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:57:01.175935 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:01.175905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79c4bccd4b-9kzkn"
Jul 16 21:57:01.305114 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:01.305080    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:57:18.183579 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.183544    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"]
Jul 16 21:57:18.190307 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.190286    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.193895 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.193876    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 21:57:18.195238 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.195216    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 21:57:18.195392 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.195369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hcdd7\""
Jul 16 21:57:18.195461 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.195399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 21:57:18.195514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.195346    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:57:18.195685 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.195663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 21:57:18.197497 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.197478    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"]
Jul 16 21:57:18.280189 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.280140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-metrics-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.280189 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.280194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/742b2f6e-8d01-4e3c-9825-6673d123e697-manager-config\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.280440 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.280229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.280440 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.280310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkbx\" (UniqueName: \"kubernetes.io/projected/742b2f6e-8d01-4e3c-9825-6673d123e697-kube-api-access-rxkbx\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.381627 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.381588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-metrics-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.381627 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.381630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/742b2f6e-8d01-4e3c-9825-6673d123e697-manager-config\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.381869 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.381656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.381869 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.381694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkbx\" (UniqueName: \"kubernetes.io/projected/742b2f6e-8d01-4e3c-9825-6673d123e697-kube-api-access-rxkbx\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.382343 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.382315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/742b2f6e-8d01-4e3c-9825-6673d123e697-manager-config\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.384515 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.384489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.384622 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.384489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/742b2f6e-8d01-4e3c-9825-6673d123e697-metrics-cert\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.398857 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.398808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkbx\" (UniqueName: \"kubernetes.io/projected/742b2f6e-8d01-4e3c-9825-6673d123e697-kube-api-access-rxkbx\") pod \"lws-controller-manager-5fcdcd6776-99jlx\" (UID: \"742b2f6e-8d01-4e3c-9825-6673d123e697\") " pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.501312 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.501284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:18.679167 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:18.679135    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"]
Jul 16 21:57:18.682689 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:57:18.682662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742b2f6e_8d01_4e3c_9825_6673d123e697.slice/crio-bc5d7d84966addd656e391308afc4bce1939c4cc86fe8de18f8586239ba5a6b2 WatchSource:0}: Error finding container bc5d7d84966addd656e391308afc4bce1939c4cc86fe8de18f8586239ba5a6b2: Status 404 returned error can't find the container with id bc5d7d84966addd656e391308afc4bce1939c4cc86fe8de18f8586239ba5a6b2
Jul 16 21:57:19.251910 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:19.251876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx" event={"ID":"742b2f6e-8d01-4e3c-9825-6673d123e697","Type":"ContainerStarted","Data":"bc5d7d84966addd656e391308afc4bce1939c4cc86fe8de18f8586239ba5a6b2"}
Jul 16 21:57:21.263115 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:21.263082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx" event={"ID":"742b2f6e-8d01-4e3c-9825-6673d123e697","Type":"ContainerStarted","Data":"82cb7b8abfebfa74501d90fdddfe18b66b7a96f811d9bdc7a95f2bc2b11ffb76"}
Jul 16 21:57:21.263493 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:21.263184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:57:21.288899 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:21.288841    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx" podStartSLOduration=0.843335967 podStartE2EDuration="3.288803166s" totalImagesPullingTime="2.445467199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:21.287436838 +0000 UTC m=+441.382948155" watchObservedRunningTime="2026-07-16 21:57:21.288803166 +0000 UTC m=+441.384314488"
Jul 16 21:57:26.326311 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.326268    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ff5d44c7b-cflp6" podUID="d9e45c16-7294-4f95-a7b1-b2d56d917286" containerName="console" containerID="cri-o://f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9" gracePeriod=15
Jul 16 21:57:26.572872 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.572842    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:57:26.670263 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670173    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670263 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670244    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670499 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670283    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670499 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670499 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670349    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfl84\" (UniqueName: \"kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670499 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670499 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670432    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config\") pod \"d9e45c16-7294-4f95-a7b1-b2d56d917286\" (UID: \"d9e45c16-7294-4f95-a7b1-b2d56d917286\") "
Jul 16 21:57:26.670757 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config" (OuterVolumeSpecName: "console-config") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:57:26.670925 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:57:26.670925 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670873    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:57:26.670925 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.670871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:57:26.672730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.672702    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84" (OuterVolumeSpecName: "kube-api-access-vfl84") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "kube-api-access-vfl84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:26.672730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.672711    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:57:26.672730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.672725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9e45c16-7294-4f95-a7b1-b2d56d917286" (UID: "d9e45c16-7294-4f95-a7b1-b2d56d917286"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771459    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-oauth-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771486    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-oauth-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771496    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-config\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771505    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-service-ca\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771513    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9e45c16-7294-4f95-a7b1-b2d56d917286-console-serving-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771514 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771521    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9e45c16-7294-4f95-a7b1-b2d56d917286-trusted-ca-bundle\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:26.771871 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:26.771531    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfl84\" (UniqueName: \"kubernetes.io/projected/d9e45c16-7294-4f95-a7b1-b2d56d917286-kube-api-access-vfl84\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:57:27.292553 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.292519    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9e45c16-7294-4f95-a7b1-b2d56d917286" containerID="f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9" exitCode=0
Jul 16 21:57:27.292746 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.292565    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ff5d44c7b-cflp6"
Jul 16 21:57:27.292746 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.292570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff5d44c7b-cflp6" event={"ID":"d9e45c16-7294-4f95-a7b1-b2d56d917286","Type":"ContainerDied","Data":"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"}
Jul 16 21:57:27.292746 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.292612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ff5d44c7b-cflp6" event={"ID":"d9e45c16-7294-4f95-a7b1-b2d56d917286","Type":"ContainerDied","Data":"3873dd7a30dd31a2955d91b1dc8085235b4928375cc79509480a22d5f2850926"}
Jul 16 21:57:27.292746 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.292632    2566 scope.go:117] "RemoveContainer" containerID="f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"
Jul 16 21:57:27.302107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.302083    2566 scope.go:117] "RemoveContainer" containerID="f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"
Jul 16 21:57:27.302424 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:57:27.302407    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9\": container with ID starting with f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9 not found: ID does not exist" containerID="f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"
Jul 16 21:57:27.302478 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.302435    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9"} err="failed to get container status \"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9\": rpc error: code = NotFound desc = could not find container \"f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9\": container with ID starting with f4cac20b3ace6c71d5da376933a217c6ea8147b1c7d742f62e4f39055c9bc3b9 not found: ID does not exist"
Jul 16 21:57:27.330639 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.330613    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:57:27.334351 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:27.334324    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ff5d44c7b-cflp6"]
Jul 16 21:57:28.319917 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:28.319874    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e45c16-7294-4f95-a7b1-b2d56d917286" path="/var/lib/kubelet/pods/d9e45c16-7294-4f95-a7b1-b2d56d917286/volumes"
Jul 16 21:57:32.269940 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:57:32.269903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5fcdcd6776-99jlx"
Jul 16 21:58:24.046896 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.046860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:24.047319 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.047175    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e45c16-7294-4f95-a7b1-b2d56d917286" containerName="console"
Jul 16 21:58:24.047319 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.047185    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e45c16-7294-4f95-a7b1-b2d56d917286" containerName="console"
Jul 16 21:58:24.047319 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.047257    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e45c16-7294-4f95-a7b1-b2d56d917286" containerName="console"
Jul 16 21:58:24.050111 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.050094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.053965 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.053945    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p86z2\""
Jul 16 21:58:24.055145 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.055124    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:58:24.055145 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.055137    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:58:24.055315 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.055190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 21:58:24.066139 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.066118    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:24.127214 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.127183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q2z\" (UniqueName: \"kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.127384 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.127222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.142672 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.142635    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:24.227585 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.227551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q2z\" (UniqueName: \"kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.227755 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.227594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.228302 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.228283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.250643 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.250618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q2z\" (UniqueName: \"kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z\") pod \"limitador-limitador-ff68ff5cb-ph894\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.360873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.360749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:24.501938 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.501907    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:24.504564 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:58:24.504535    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c6ab92_cab6_4d38_a954_e2be56a48ef3.slice/crio-8b92a56e68092baddc15fb979821bf8a01c323197ba01cc51b1360c821d70cf3 WatchSource:0}: Error finding container 8b92a56e68092baddc15fb979821bf8a01c323197ba01cc51b1360c821d70cf3: Status 404 returned error can't find the container with id 8b92a56e68092baddc15fb979821bf8a01c323197ba01cc51b1360c821d70cf3
Jul 16 21:58:24.542781 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:24.542747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" event={"ID":"50c6ab92-cab6-4d38-a954-e2be56a48ef3","Type":"ContainerStarted","Data":"8b92a56e68092baddc15fb979821bf8a01c323197ba01cc51b1360c821d70cf3"}
Jul 16 21:58:25.319524 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.319464    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:25.322983 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.322960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:25.328030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.327983    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hjvdh\""
Jul 16 21:58:25.333832 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.333795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:25.438471 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.438426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzlgx\" (UniqueName: \"kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx\") pod \"authorino-8df697c9f-cgx8d\" (UID: \"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7\") " pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:25.539598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.539562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzlgx\" (UniqueName: \"kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx\") pod \"authorino-8df697c9f-cgx8d\" (UID: \"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7\") " pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:25.549669 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.549639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzlgx\" (UniqueName: \"kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx\") pod \"authorino-8df697c9f-cgx8d\" (UID: \"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7\") " pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:25.635460 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.635384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:25.793550 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:25.793519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:25.795668 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:58:25.795638    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aa269ba_c75b_4b56_bbe4_6ee66c1c3de7.slice/crio-59f8bcddcc53310c41b86dab2bdc37a4675be04170a09d9bd3150d61dab8ff39 WatchSource:0}: Error finding container 59f8bcddcc53310c41b86dab2bdc37a4675be04170a09d9bd3150d61dab8ff39: Status 404 returned error can't find the container with id 59f8bcddcc53310c41b86dab2bdc37a4675be04170a09d9bd3150d61dab8ff39
Jul 16 21:58:26.557381 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:26.557341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cgx8d" event={"ID":"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7","Type":"ContainerStarted","Data":"59f8bcddcc53310c41b86dab2bdc37a4675be04170a09d9bd3150d61dab8ff39"}
Jul 16 21:58:29.574317 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:29.574276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" event={"ID":"50c6ab92-cab6-4d38-a954-e2be56a48ef3","Type":"ContainerStarted","Data":"dc804d2165643c8f495f07a94e9e279111b646a0ca2d26ff037fd3c0fb5161c4"}
Jul 16 21:58:29.574849 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:29.574432    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:29.575732 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:29.575708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cgx8d" event={"ID":"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7","Type":"ContainerStarted","Data":"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"}
Jul 16 21:58:29.596239 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:29.596188    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" podStartSLOduration=1.45132285 podStartE2EDuration="5.596171274s" totalImagesPullingTime="4.144848424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:29.59514401 +0000 UTC m=+509.690655333" watchObservedRunningTime="2026-07-16 21:58:29.596171274 +0000 UTC m=+509.691682632"
Jul 16 21:58:29.611687 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:29.611637    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-cgx8d" podStartSLOduration=1.755422482 podStartE2EDuration="4.61162179s" totalImagesPullingTime="2.856199308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:29.610125453 +0000 UTC m=+509.705636769" watchObservedRunningTime="2026-07-16 21:58:29.61162179 +0000 UTC m=+509.707133106"
Jul 16 21:58:39.997627 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:39.997590    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:39.998140 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:39.997801    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" podUID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" containerName="limitador" containerID="cri-o://dc804d2165643c8f495f07a94e9e279111b646a0ca2d26ff037fd3c0fb5161c4" gracePeriod=30
Jul 16 21:58:39.998580 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:39.998557    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:40.624125 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.624045    2566 generic.go:356] "Generic (PLEG): container finished" podID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" containerID="dc804d2165643c8f495f07a94e9e279111b646a0ca2d26ff037fd3c0fb5161c4" exitCode=0
Jul 16 21:58:40.624125 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.624087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" event={"ID":"50c6ab92-cab6-4d38-a954-e2be56a48ef3","Type":"ContainerDied","Data":"dc804d2165643c8f495f07a94e9e279111b646a0ca2d26ff037fd3c0fb5161c4"}
Jul 16 21:58:40.948582 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.948554    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:40.976600 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.976569    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5q2z\" (UniqueName: \"kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z\") pod \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") "
Jul 16 21:58:40.976782 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.976670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file\") pod \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\" (UID: \"50c6ab92-cab6-4d38-a954-e2be56a48ef3\") "
Jul 16 21:58:40.977118 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.977085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file" (OuterVolumeSpecName: "config-file") pod "50c6ab92-cab6-4d38-a954-e2be56a48ef3" (UID: "50c6ab92-cab6-4d38-a954-e2be56a48ef3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:58:40.979030 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:40.978996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z" (OuterVolumeSpecName: "kube-api-access-w5q2z") pod "50c6ab92-cab6-4d38-a954-e2be56a48ef3" (UID: "50c6ab92-cab6-4d38-a954-e2be56a48ef3"). InnerVolumeSpecName "kube-api-access-w5q2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:58:41.077774 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.077738    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/50c6ab92-cab6-4d38-a954-e2be56a48ef3-config-file\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:58:41.077774 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.077766    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5q2z\" (UniqueName: \"kubernetes.io/projected/50c6ab92-cab6-4d38-a954-e2be56a48ef3-kube-api-access-w5q2z\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:58:41.629887 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.629855    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894"
Jul 16 21:58:41.630104 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.629854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-ph894" event={"ID":"50c6ab92-cab6-4d38-a954-e2be56a48ef3","Type":"ContainerDied","Data":"8b92a56e68092baddc15fb979821bf8a01c323197ba01cc51b1360c821d70cf3"}
Jul 16 21:58:41.630104 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.629993    2566 scope.go:117] "RemoveContainer" containerID="dc804d2165643c8f495f07a94e9e279111b646a0ca2d26ff037fd3c0fb5161c4"
Jul 16 21:58:41.651618 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.651590    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:41.653884 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:41.653862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-ph894"]
Jul 16 21:58:42.319699 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:42.319665    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" path="/var/lib/kubelet/pods/50c6ab92-cab6-4d38-a954-e2be56a48ef3/volumes"
Jul 16 21:58:52.311717 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.311642    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-w4lkk"]
Jul 16 21:58:52.312096 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.312018    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" containerName="limitador"
Jul 16 21:58:52.312096 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.312030    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" containerName="limitador"
Jul 16 21:58:52.312096 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.312087    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c6ab92-cab6-4d38-a954-e2be56a48ef3" containerName="limitador"
Jul 16 21:58:52.315196 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.315168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.318233 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.318210    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 21:58:52.332692 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.332673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-w4lkk"]
Jul 16 21:58:52.354647 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.354623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-tls-cert\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.354799 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.354650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smj4f\" (UniqueName: \"kubernetes.io/projected/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-kube-api-access-smj4f\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.455288 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.455256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-tls-cert\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.455288 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.455297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smj4f\" (UniqueName: \"kubernetes.io/projected/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-kube-api-access-smj4f\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.457868 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.457846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-tls-cert\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.475054 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.475031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smj4f\" (UniqueName: \"kubernetes.io/projected/c3de1e90-90b7-4d7b-8189-afbe3e742cb4-kube-api-access-smj4f\") pod \"authorino-7c77957575-w4lkk\" (UID: \"c3de1e90-90b7-4d7b-8189-afbe3e742cb4\") " pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.624873 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.624762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-w4lkk"
Jul 16 21:58:52.782314 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:52.782287    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-w4lkk"]
Jul 16 21:58:52.784758 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:58:52.784726    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3de1e90_90b7_4d7b_8189_afbe3e742cb4.slice/crio-fc9f26ef9ef4f29660b0e58ce91ec9f3ba440b5da685ed330c77246cd4d17ade WatchSource:0}: Error finding container fc9f26ef9ef4f29660b0e58ce91ec9f3ba440b5da685ed330c77246cd4d17ade: Status 404 returned error can't find the container with id fc9f26ef9ef4f29660b0e58ce91ec9f3ba440b5da685ed330c77246cd4d17ade
Jul 16 21:58:53.688078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.687985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-w4lkk" event={"ID":"c3de1e90-90b7-4d7b-8189-afbe3e742cb4","Type":"ContainerStarted","Data":"01e4898a822ec61dfa522ab3286811a86c2450485d91c9701eb40a6782a35c82"}
Jul 16 21:58:53.688078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.688029    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-w4lkk" event={"ID":"c3de1e90-90b7-4d7b-8189-afbe3e742cb4","Type":"ContainerStarted","Data":"fc9f26ef9ef4f29660b0e58ce91ec9f3ba440b5da685ed330c77246cd4d17ade"}
Jul 16 21:58:53.707416 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.707365    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-w4lkk" podStartSLOduration=1.1616644680000001 podStartE2EDuration="1.707348377s" totalImagesPullingTime="545.683909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:53.706188933 +0000 UTC m=+533.801700250" watchObservedRunningTime="2026-07-16 21:58:53.707348377 +0000 UTC m=+533.802859694"
Jul 16 21:58:53.735169 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.735132    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:53.735384 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.735357    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-cgx8d" podUID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" containerName="authorino" containerID="cri-o://72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d" gracePeriod=30
Jul 16 21:58:53.997011 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:53.996986    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:54.069565 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.069519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzlgx\" (UniqueName: \"kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx\") pod \"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7\" (UID: \"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7\") "
Jul 16 21:58:54.071805 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.071779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx" (OuterVolumeSpecName: "kube-api-access-rzlgx") pod "2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" (UID: "2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7"). InnerVolumeSpecName "kube-api-access-rzlgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:58:54.171132 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.171094    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzlgx\" (UniqueName: \"kubernetes.io/projected/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7-kube-api-access-rzlgx\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 21:58:54.693534 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.693439    2566 generic.go:356] "Generic (PLEG): container finished" podID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" containerID="72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d" exitCode=0
Jul 16 21:58:54.693534 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.693480    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cgx8d"
Jul 16 21:58:54.693534 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.693525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cgx8d" event={"ID":"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7","Type":"ContainerDied","Data":"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"}
Jul 16 21:58:54.694107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.693567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cgx8d" event={"ID":"2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7","Type":"ContainerDied","Data":"59f8bcddcc53310c41b86dab2bdc37a4675be04170a09d9bd3150d61dab8ff39"}
Jul 16 21:58:54.694107 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.693593    2566 scope.go:117] "RemoveContainer" containerID="72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"
Jul 16 21:58:54.701730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.701712    2566 scope.go:117] "RemoveContainer" containerID="72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"
Jul 16 21:58:54.702001 ip-10-0-143-100 kubenswrapper[2566]: E0716 21:58:54.701983    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d\": container with ID starting with 72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d not found: ID does not exist" containerID="72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"
Jul 16 21:58:54.702075 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.702012    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d"} err="failed to get container status \"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d\": rpc error: code = NotFound desc = could not find container \"72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d\": container with ID starting with 72444912abe89ae79c771b62cace419f7240e74adb485983447735e5aa2bef2d not found: ID does not exist"
Jul 16 21:58:54.724967 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.724936    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:54.734618 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:54.734592    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cgx8d"]
Jul 16 21:58:56.319078 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:58:56.319042    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" path="/var/lib/kubelet/pods/2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7/volumes"
Jul 16 21:59:31.232777 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.232737    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wsrth"]
Jul 16 21:59:31.233408 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.233233    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" containerName="authorino"
Jul 16 21:59:31.233408 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.233252    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" containerName="authorino"
Jul 16 21:59:31.233408 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.233367    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2aa269ba-c75b-4b56-bbe4-6ee66c1c3de7" containerName="authorino"
Jul 16 21:59:31.236435 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.236413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.241217 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.241190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-7d8rh\""
Jul 16 21:59:31.241354 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.241236    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 16 21:59:31.267251 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.267206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wsrth"]
Jul 16 21:59:31.396135 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.396101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-tls-certs\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.396329 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.396169    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4b9\" (UniqueName: \"kubernetes.io/projected/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-kube-api-access-bp4b9\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.496998 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.496901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-tls-certs\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.497155 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.497010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4b9\" (UniqueName: \"kubernetes.io/projected/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-kube-api-access-bp4b9\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.499486 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.499463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-tls-certs\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.510008 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.509977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4b9\" (UniqueName: \"kubernetes.io/projected/33f9d154-fd1f-45d9-a7a9-ebbd9709be15-kube-api-access-bp4b9\") pod \"model-serving-api-fd65d7d6-wsrth\" (UID: \"33f9d154-fd1f-45d9-a7a9-ebbd9709be15\") " pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.546830 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.546777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:31.704156 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.704128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wsrth"]
Jul 16 21:59:31.707221 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:59:31.707192    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f9d154_fd1f_45d9_a7a9_ebbd9709be15.slice/crio-aa5b38e1d53a26ce8340d7588762b323a3ae480cf06ab8af1b42ebdaaa6c189f WatchSource:0}: Error finding container aa5b38e1d53a26ce8340d7588762b323a3ae480cf06ab8af1b42ebdaaa6c189f: Status 404 returned error can't find the container with id aa5b38e1d53a26ce8340d7588762b323a3ae480cf06ab8af1b42ebdaaa6c189f
Jul 16 21:59:31.862730 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:31.862649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wsrth" event={"ID":"33f9d154-fd1f-45d9-a7a9-ebbd9709be15","Type":"ContainerStarted","Data":"aa5b38e1d53a26ce8340d7588762b323a3ae480cf06ab8af1b42ebdaaa6c189f"}
Jul 16 21:59:34.878156 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:34.878117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wsrth" event={"ID":"33f9d154-fd1f-45d9-a7a9-ebbd9709be15","Type":"ContainerStarted","Data":"c20939009644cadcf462ee91bdf6db2f5f78a46c3ddbec4573fd6be3785ecb34"}
Jul 16 21:59:34.878640 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:34.878231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:34.896360 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:34.896312    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wsrth" podStartSLOduration=1.5458538339999999 podStartE2EDuration="3.896295475s" totalImagesPullingTime="2.350441641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:34.895251721 +0000 UTC m=+574.990763038" watchObservedRunningTime="2026-07-16 21:59:34.896295475 +0000 UTC m=+574.991806846"
Jul 16 21:59:45.886598 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:45.886563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wsrth"
Jul 16 21:59:46.944150 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:46.944112    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cf58t"]
Jul 16 21:59:46.950747 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:46.950719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cf58t"
Jul 16 21:59:46.954302 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:46.954284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-v4cv8\""
Jul 16 21:59:46.954529 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:46.954502    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 21:59:46.957695 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:46.957669    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cf58t"]
Jul 16 21:59:47.031273 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.031238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w72w\" (UniqueName: \"kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.031438 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.031351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.132432 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.132401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.132583 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.132439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w72w\" (UniqueName: \"kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.132758 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.132738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.142266 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.142240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w72w\" (UniqueName: \"kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w\") pod \"s3-init-cf58t\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") " pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.260783 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.260749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cf58t"
Jul 16 21:59:47.398137 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.397984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cf58t"]
Jul 16 21:59:47.400229 ip-10-0-143-100 kubenswrapper[2566]: W0716 21:59:47.400201    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99fa3c05_0f9a_496b_9110_7baf5c38c04e.slice/crio-cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9 WatchSource:0}: Error finding container cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9: Status 404 returned error can't find the container with id cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9
Jul 16 21:59:47.938257 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:47.938223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerStarted","Data":"cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9"}
Jul 16 21:59:50.958859 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:50.958811    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerStarted","Data":"c8daefadb5e7beabcedc5251c86c6b64291904c859f9b7d5299bc32b703b9d36"}
Jul 16 21:59:51.964361 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:51.964326    2566 generic.go:356] "Generic (PLEG): container finished" podID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerID="c8daefadb5e7beabcedc5251c86c6b64291904c859f9b7d5299bc32b703b9d36" exitCode=0
Jul 16 21:59:51.964759 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:51.964420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerDied","Data":"c8daefadb5e7beabcedc5251c86c6b64291904c859f9b7d5299bc32b703b9d36"}
Jul 16 21:59:56.993095 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:56.993063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerStarted","Data":"bd53dfd3b656d02effa3a7e428489d897e36c907290f1b1e066c32f884376bfb"}
Jul 16 21:59:57.026420 ip-10-0-143-100 kubenswrapper[2566]: I0716 21:59:57.026370    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cf58t" podStartSLOduration=2.228257259 podStartE2EDuration="11.026354442s" totalImagesPullingTime="7.750068344s" totalInitContainerRuntime="1.048028839s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:59:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:59:57.025864972 +0000 UTC m=+597.121376289" watchObservedRunningTime="2026-07-16 21:59:57.026354442 +0000 UTC m=+597.121865759"
Jul 16 22:00:00.262645 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:00:00.262618    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:00:00.263116 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:00:00.262652    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:01:02.293664 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:02.293629    2566 generic.go:356] "Generic (PLEG): container finished" podID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerID="bd53dfd3b656d02effa3a7e428489d897e36c907290f1b1e066c32f884376bfb" exitCode=0
Jul 16 22:01:02.294091 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:02.293672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerDied","Data":"bd53dfd3b656d02effa3a7e428489d897e36c907290f1b1e066c32f884376bfb"}
Jul 16 22:01:03.441984 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.441958    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cf58t"
Jul 16 22:01:03.484143 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.484111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache\") pod \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") "
Jul 16 22:01:03.484304 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.484183    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w72w\" (UniqueName: \"kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w\") pod \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\" (UID: \"99fa3c05-0f9a-496b-9110-7baf5c38c04e\") "
Jul 16 22:01:03.486439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.486410    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w" (OuterVolumeSpecName: "kube-api-access-4w72w") pod "99fa3c05-0f9a-496b-9110-7baf5c38c04e" (UID: "99fa3c05-0f9a-496b-9110-7baf5c38c04e"). InnerVolumeSpecName "kube-api-access-4w72w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:01:03.502923 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.502891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache" (OuterVolumeSpecName: "model-cache") pod "99fa3c05-0f9a-496b-9110-7baf5c38c04e" (UID: "99fa3c05-0f9a-496b-9110-7baf5c38c04e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:01:03.585226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.585144    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w72w\" (UniqueName: \"kubernetes.io/projected/99fa3c05-0f9a-496b-9110-7baf5c38c04e-kube-api-access-4w72w\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:03.585226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:03.585173    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99fa3c05-0f9a-496b-9110-7baf5c38c04e-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:04.305260 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:04.305233    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cf58t"
Jul 16 22:01:04.305260 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:04.305251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cf58t" event={"ID":"99fa3c05-0f9a-496b-9110-7baf5c38c04e","Type":"ContainerDied","Data":"cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9"}
Jul 16 22:01:04.305480 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:04.305284    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5f21dcd441e9e90aa3e454e5415e23924dd6125861244517d44be2abf388e9"
Jul 16 22:01:25.168811 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.168771    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:25.169344 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.169326    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerName="s3-init"
Jul 16 22:01:25.169344 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.169345    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerName="s3-init"
Jul 16 22:01:25.169459 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.169373    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerName="download-hf-model"
Jul 16 22:01:25.169459 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.169384    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerName="download-hf-model"
Jul 16 22:01:25.169550 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.169474    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99fa3c05-0f9a-496b-9110-7baf5c38c04e" containerName="s3-init"
Jul 16 22:01:25.172464 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.172439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.175412 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.175384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 16 22:01:25.175537 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.175443    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:01:25.177338 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.177318    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:01:25.177442 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.177357    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:01:25.184149 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.184129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:25.272692 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.272910 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.272910 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.272910 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272900    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.273084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.273084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.272967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xw7g\" (UniqueName: \"kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373643 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373643 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373643 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373768 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373690    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373768 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xw7g\" (UniqueName: \"kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373925 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.373990 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.373973    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.374097 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.374076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.375899 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.375881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.376233 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.376211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.382947 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.382923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xw7g\" (UniqueName: \"kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.483275 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.483174    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:25.653423 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.653306    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:25.656097 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:01:25.656070    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b73cc6_4d71_4a72_bde6_f5f6877b03cf.slice/crio-bbccb8a9fdb802c733f65491d2e5857ec7c810fb6f0637b029d12840f5c88afd WatchSource:0}: Error finding container bbccb8a9fdb802c733f65491d2e5857ec7c810fb6f0637b029d12840f5c88afd: Status 404 returned error can't find the container with id bbccb8a9fdb802c733f65491d2e5857ec7c810fb6f0637b029d12840f5c88afd
Jul 16 22:01:25.658167 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:25.658150    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:01:26.406765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:26.406727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" event={"ID":"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf","Type":"ContainerStarted","Data":"bbccb8a9fdb802c733f65491d2e5857ec7c810fb6f0637b029d12840f5c88afd"}
Jul 16 22:01:27.412642 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:27.412606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" event={"ID":"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf","Type":"ContainerStarted","Data":"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"}
Jul 16 22:01:27.433471 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:27.433421    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" podStartSLOduration=1.365673784 podStartE2EDuration="2.433406667s" totalImagesPullingTime="1.067732883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:27.430960767 +0000 UTC m=+687.526472085" watchObservedRunningTime="2026-07-16 22:01:27.433406667 +0000 UTC m=+687.528917984"
Jul 16 22:01:27.483935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:27.483897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:35.483762 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:35.483724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:35.495973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:35.495941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:36.464335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:36.464306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:38.717107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.717072    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:01:38.720560 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.720541    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.723459 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.723439    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:01:38.723604 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.723476    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 16 22:01:38.745712 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.745670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:01:38.788208 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788208 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788420 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788420 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drwnq\" (UniqueName: \"kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788420 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788420 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.788547 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.788426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889330 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889330 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889336    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889375    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drwnq\" (UniqueName: \"kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889932 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889932 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.889932 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.890088 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.889938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.891906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.891886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.892036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.892021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:38.901597 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:38.901564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drwnq\" (UniqueName: \"kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq\") pod \"precise-prefix-cache-test-kserve-7597885744-t6hgf\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:39.031946 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.031850    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:39.177164 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.177140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:01:39.179406 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:01:39.179378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e403c51_0e76_4322_bb7d_6481a5c5d444.slice/crio-138283c85af21adec45ed8a8ecfa38ec5d176204b317ab3d39f90848e2726c24 WatchSource:0}: Error finding container 138283c85af21adec45ed8a8ecfa38ec5d176204b317ab3d39f90848e2726c24: Status 404 returned error can't find the container with id 138283c85af21adec45ed8a8ecfa38ec5d176204b317ab3d39f90848e2726c24
Jul 16 22:01:39.278391 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.278344    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:01:39.281888 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.281864    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.284755 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.284680    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-rg5lb\""
Jul 16 22:01:39.295188 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.295152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:01:39.300647 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.300799 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.300799 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.300799 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.300799 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.300988 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.300833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknct\" (UniqueName: \"kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.401592 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.401784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.401784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.401927 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401858    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.401927 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknct\" (UniqueName: \"kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.402036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.401955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.402088 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.402043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.402128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.402080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.402222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.402181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.402333 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.402309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.404418 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.404395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.411578 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.411558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknct\" (UniqueName: \"kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct\") pod \"precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.470573 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.470537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerStarted","Data":"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"}
Jul 16 22:01:39.470768 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.470608    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerStarted","Data":"138283c85af21adec45ed8a8ecfa38ec5d176204b317ab3d39f90848e2726c24"}
Jul 16 22:01:39.591784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.591689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:01:39.763114 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:39.763090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:01:39.765293 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:01:39.765265    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517feab4_224b_4309_9286_c8cfe4236e75.slice/crio-0fabb5e8b145f42cd680b716a51cc3432556baff2c68519d027d38f90851307e WatchSource:0}: Error finding container 0fabb5e8b145f42cd680b716a51cc3432556baff2c68519d027d38f90851307e: Status 404 returned error can't find the container with id 0fabb5e8b145f42cd680b716a51cc3432556baff2c68519d027d38f90851307e
Jul 16 22:01:40.481001 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:40.480960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"}
Jul 16 22:01:40.481200 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:40.481021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"0fabb5e8b145f42cd680b716a51cc3432556baff2c68519d027d38f90851307e"}
Jul 16 22:01:41.487216 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:41.487182    2566 generic.go:356] "Generic (PLEG): container finished" podID="517feab4-224b-4309-9286-c8cfe4236e75" containerID="7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d" exitCode=0
Jul 16 22:01:41.487609 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:41.487259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"}
Jul 16 22:01:42.021678 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:42.021643    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:42.493880 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:42.493840    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerID="c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320" exitCode=0
Jul 16 22:01:42.494246 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:42.493915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerDied","Data":"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"}
Jul 16 22:01:43.503750 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:43.503702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerStarted","Data":"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"}
Jul 16 22:01:43.526949 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:43.526887    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" podStartSLOduration=2.499266609 podStartE2EDuration="5.52685411s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.027587501s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:43.525239993 +0000 UTC m=+703.620751321" watchObservedRunningTime="2026-07-16 22:01:43.52685411 +0000 UTC m=+703.622365429"
Jul 16 22:01:44.032340 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:44.031998    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:45.516789 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:45.516712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"1eeb32109195a981e5211025cf4aca9e3064f3c430abe2e945c8da5e566d9f03"}
Jul 16 22:01:49.032207 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:49.032031    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:49.047603 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:49.047580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:49.554554 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:49.554515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:01:55.575386 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:55.575360    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/0.log"
Jul 16 22:01:55.575847 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:55.575756    2566 generic.go:356] "Generic (PLEG): container finished" podID="517feab4-224b-4309-9286-c8cfe4236e75" containerID="1eeb32109195a981e5211025cf4aca9e3064f3c430abe2e945c8da5e566d9f03" exitCode=1
Jul 16 22:01:55.575931 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:55.575863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"1eeb32109195a981e5211025cf4aca9e3064f3c430abe2e945c8da5e566d9f03"}
Jul 16 22:01:57.046186 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.046145    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" podUID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" containerName="main" containerID="cri-o://bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce" gracePeriod=45
Jul 16 22:01:57.355105 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.355066    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:57.477934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.477896    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.477934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.477938    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.478181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.477958    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xw7g\" (UniqueName: \"kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.478181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.477993    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.478181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478014    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.478181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478088    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs\") pod \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\" (UID: \"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf\") "
Jul 16 22:01:57.478365 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478237    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:01:57.478365 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home" (OuterVolumeSpecName: "home") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:01:57.478365 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:01:57.478523 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478363    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.478523 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.478379    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.480362 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.480336    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm" (OuterVolumeSpecName: "dshm") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:01:57.480362 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.480345    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:01:57.480504 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.480370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g" (OuterVolumeSpecName: "kube-api-access-9xw7g") pod "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" (UID: "b4b73cc6-4d71-4a72-bde6-f5f6877b03cf"). InnerVolumeSpecName "kube-api-access-9xw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:01:57.578939 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.578852    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xw7g\" (UniqueName: \"kubernetes.io/projected/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-kube-api-access-9xw7g\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.578939 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.578882    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.578939 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.578895    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.578939 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.578906    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:01:57.593469 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.593433    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" containerID="bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce" exitCode=0
Jul 16 22:01:57.593635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.593479    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"
Jul 16 22:01:57.593635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.593516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" event={"ID":"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf","Type":"ContainerDied","Data":"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"}
Jul 16 22:01:57.593635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.593569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc" event={"ID":"b4b73cc6-4d71-4a72-bde6-f5f6877b03cf","Type":"ContainerDied","Data":"bbccb8a9fdb802c733f65491d2e5857ec7c810fb6f0637b029d12840f5c88afd"}
Jul 16 22:01:57.593635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.593589    2566 scope.go:117] "RemoveContainer" containerID="bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"
Jul 16 22:01:57.602765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.602750    2566 scope.go:117] "RemoveContainer" containerID="bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"
Jul 16 22:01:57.603070 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:01:57.603051    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce\": container with ID starting with bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce not found: ID does not exist" containerID="bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"
Jul 16 22:01:57.603121 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.603079    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce"} err="failed to get container status \"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce\": rpc error: code = NotFound desc = could not find container \"bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce\": container with ID starting with bb79ef34e82ebcfb8776e9b298e582b2685f8bd872a5bca0851033c3f6e330ce not found: ID does not exist"
Jul 16 22:01:57.837275 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.837199    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:57.850905 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:57.850871    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5xdnmc"]
Jul 16 22:01:58.320020 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:01:58.319984    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" path="/var/lib/kubelet/pods/b4b73cc6-4d71-4a72-bde6-f5f6877b03cf/volumes"
Jul 16 22:02:26.764726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:26.764692    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/0.log"
Jul 16 22:02:26.765224 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:26.765201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"}
Jul 16 22:02:26.765520 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:26.765502    2566 scope.go:117] "RemoveContainer" containerID="1eeb32109195a981e5211025cf4aca9e3064f3c430abe2e945c8da5e566d9f03"
Jul 16 22:02:27.774494 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:27.774401    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/0.log"
Jul 16 22:02:27.776322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:27.775349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"}
Jul 16 22:02:27.776322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:27.776261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:27.801888 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:27.801801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podStartSLOduration=3.446497564 podStartE2EDuration="48.801779858s" totalImagesPullingTime="44.504901005s" totalInitContainerRuntime="850.381289ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:27.800191489 +0000 UTC m=+747.895702831" watchObservedRunningTime="2026-07-16 22:02:27.801779858 +0000 UTC m=+747.897291175"
Jul 16 22:02:29.002502 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.002455    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:02:29.003075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.003054    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" containerName="main"
Jul 16 22:02:29.003144 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.003080    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" containerName="main"
Jul 16 22:02:29.003208 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.003197    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4b73cc6-4d71-4a72-bde6-f5f6877b03cf" containerName="main"
Jul 16 22:02:29.006898 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.006875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.011317 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.011293    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-bvn7s\""
Jul 16 22:02:29.011491 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.011295    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 16 22:02:29.017361 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.017333    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:02:29.075593 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.075557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bpp\" (UniqueName: \"kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.075759 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.075615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.075759 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.075643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.075759 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.075662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.176735 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.176701    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.176735 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.176738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.177017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.176776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.177017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.176885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bpp\" (UniqueName: \"kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.177152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.177128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.177236 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.177209    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.179320 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.179304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.185675 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.185648    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bpp\" (UniqueName: \"kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp\") pod \"auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.318188 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.318099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:29.485235 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.485159    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:02:29.496496 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:02:29.496117    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ea3b40_ca2e_4ec2_9cb7_439a2ac1f756.slice/crio-ece0318b5225735e6f3166df0fc903ed3068228ef3b192bad8fb5fe8d050f85c WatchSource:0}: Error finding container ece0318b5225735e6f3166df0fc903ed3068228ef3b192bad8fb5fe8d050f85c: Status 404 returned error can't find the container with id ece0318b5225735e6f3166df0fc903ed3068228ef3b192bad8fb5fe8d050f85c
Jul 16 22:02:29.592434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.592343    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:29.592434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.592426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:29.795892 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.795797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerStarted","Data":"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"}
Jul 16 22:02:29.796073 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:29.795949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerStarted","Data":"ece0318b5225735e6f3166df0fc903ed3068228ef3b192bad8fb5fe8d050f85c"}
Jul 16 22:02:30.803093 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:30.803062    2566 generic.go:356] "Generic (PLEG): container finished" podID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerID="a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b" exitCode=0
Jul 16 22:02:30.803535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:30.803146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerDied","Data":"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"}
Jul 16 22:02:31.809844 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:31.809777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerStarted","Data":"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"}
Jul 16 22:02:31.810389 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:31.809916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:02:31.833027 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:31.832971    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" podStartSLOduration=3.075803773 podStartE2EDuration="3.832954098s" totalImagesPullingTime="0s" totalInitContainerRuntime="757.150325ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:31.831447408 +0000 UTC m=+751.926958726" watchObservedRunningTime="2026-07-16 22:02:31.832954098 +0000 UTC m=+751.928465415"
Jul 16 22:02:37.842765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.842730    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:02:37.843396 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.843377    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/0.log"
Jul 16 22:02:37.843689 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.843668    2566 generic.go:356] "Generic (PLEG): container finished" podID="517feab4-224b-4309-9286-c8cfe4236e75" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2" exitCode=1
Jul 16 22:02:37.843763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.843743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"}
Jul 16 22:02:37.843833 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.843806    2566 scope.go:117] "RemoveContainer" containerID="1eeb32109195a981e5211025cf4aca9e3064f3c430abe2e945c8da5e566d9f03"
Jul 16 22:02:37.844272 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:37.844254    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:02:37.844536 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:02:37.844515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_kserve-ci-e2e-test(517feab4-224b-4309-9286-c8cfe4236e75)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75"
Jul 16 22:02:38.849971 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:38.849935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:02:39.592918 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:39.592878    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:39.593354 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:39.593332    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:02:39.593537 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:02:39.593520    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_kserve-ci-e2e-test(517feab4-224b-4309-9286-c8cfe4236e75)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75"
Jul 16 22:02:39.595671 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:39.595646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:39.856412 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:39.856332    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:02:39.856758 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:02:39.856597    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_kserve-ci-e2e-test(517feab4-224b-4309-9286-c8cfe4236e75)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75"
Jul 16 22:02:39.857362 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:39.857345    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:40.864485 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:40.864450    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:02:40.864891 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:02:40.864690    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_kserve-ci-e2e-test(517feab4-224b-4309-9286-c8cfe4236e75)\"" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75"
Jul 16 22:02:54.314070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:54.314034    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:02:54.931153 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:54.931126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:02:54.931892 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:54.931867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerStarted","Data":"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"}
Jul 16 22:02:54.932115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:02:54.932096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:02:59.593004 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:02:59.592976    2566 logging.go:55] [core] [Channel #26 SubChannel #27]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:9003", ServerName: "10.134.0.39:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:9003: connect: connection refused"
Jul 16 22:03:00.592293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:00.592246    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:9003\" within 1s: context deadline exceeded"
Jul 16 22:03:02.817555 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:02.817519    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:03:09.593591 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:09.593549    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:03:25.939296 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:25.939222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:03:27.759767 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:27.759732    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:03:27.760177 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:27.759996    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer" containerID="cri-o://21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03" gracePeriod=60
Jul 16 22:03:27.769037 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:27.769006    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:03:29.857202 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:29.857156    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.39:8082/healthz\": dial tcp 10.134.0.39:8082: connect: connection refused"
Jul 16 22:03:30.111278 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:30.111203    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:03:30.111922 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:30.111897    2566 generic.go:356] "Generic (PLEG): container finished" podID="517feab4-224b-4309-9286-c8cfe4236e75" containerID="21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03" exitCode=0
Jul 16 22:03:30.112021 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:30.111968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"}
Jul 16 22:03:35.939374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:35.939328    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:03:39.857001 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:39.856962    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.39:8082/healthz\": dial tcp 10.134.0.39:8082: connect: connection refused"
Jul 16 22:03:42.789718 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:42.789655    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="main" containerID="cri-o://6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7" gracePeriod=45
Jul 16 22:03:42.792405 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:42.792367    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main" containerID="cri-o://14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e" gracePeriod=45
Jul 16 22:03:43.071397 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.071378    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:03:43.074659 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.074638    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:03:43.076598 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.076580    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:03:43.175878 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.175841    2566 generic.go:356] "Generic (PLEG): container finished" podID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerID="6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7" exitCode=0
Jul 16 22:03:43.176070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.175887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"
Jul 16 22:03:43.176070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.175922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerDied","Data":"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"}
Jul 16 22:03:43.176070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.175977    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf" event={"ID":"3e403c51-0e76-4322-bb7d-6481a5c5d444","Type":"ContainerDied","Data":"138283c85af21adec45ed8a8ecfa38ec5d176204b317ab3d39f90848e2726c24"}
Jul 16 22:03:43.176070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.176004    2566 scope.go:117] "RemoveContainer" containerID="6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"
Jul 16 22:03:43.177665 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.177648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm_517feab4-224b-4309-9286-c8cfe4236e75/main/1.log"
Jul 16 22:03:43.178344 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.178320    2566 generic.go:356] "Generic (PLEG): container finished" podID="517feab4-224b-4309-9286-c8cfe4236e75" containerID="14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e" exitCode=0
Jul 16 22:03:43.178411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.178379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"}
Jul 16 22:03:43.178468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.178427    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"
Jul 16 22:03:43.178468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.178432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm" event={"ID":"517feab4-224b-4309-9286-c8cfe4236e75","Type":"ContainerDied","Data":"0fabb5e8b145f42cd680b716a51cc3432556baff2c68519d027d38f90851307e"}
Jul 16 22:03:43.184969 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.184951    2566 scope.go:117] "RemoveContainer" containerID="c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"
Jul 16 22:03:43.193031 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.193016    2566 scope.go:117] "RemoveContainer" containerID="6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"
Jul 16 22:03:43.193292 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.193274    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7\": container with ID starting with 6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7 not found: ID does not exist" containerID="6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"
Jul 16 22:03:43.193350 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.193300    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7"} err="failed to get container status \"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7\": rpc error: code = NotFound desc = could not find container \"6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7\": container with ID starting with 6f35bbf38c74bda13ea80acfabec65eaa714596a23064775553f0b96e52e0fc7 not found: ID does not exist"
Jul 16 22:03:43.193350 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.193315    2566 scope.go:117] "RemoveContainer" containerID="c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"
Jul 16 22:03:43.193523 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.193506    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320\": container with ID starting with c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320 not found: ID does not exist" containerID="c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"
Jul 16 22:03:43.193586 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.193534    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320"} err="failed to get container status \"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320\": rpc error: code = NotFound desc = could not find container \"c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320\": container with ID starting with c85c6a695eadf5063b1f4b6299fc61c49c0304efcb14fa1ea45582c66e844320 not found: ID does not exist"
Jul 16 22:03:43.193586 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.193561    2566 scope.go:117] "RemoveContainer" containerID="14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"
Jul 16 22:03:43.201144 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.201130    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:03:43.208378 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.208363    2566 scope.go:117] "RemoveContainer" containerID="21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"
Jul 16 22:03:43.215659 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.215640    2566 scope.go:117] "RemoveContainer" containerID="7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"
Jul 16 22:03:43.223180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223165    2566 scope.go:117] "RemoveContainer" containerID="14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"
Jul 16 22:03:43.223459 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.223442    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e\": container with ID starting with 14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e not found: ID does not exist" containerID="14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"
Jul 16 22:03:43.223502 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223467    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e"} err="failed to get container status \"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e\": rpc error: code = NotFound desc = could not find container \"14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e\": container with ID starting with 14a6dc76a43ab4919ee0dc43b7a9db46e59d2b6bedddda37f8e33ac99e2a035e not found: ID does not exist"
Jul 16 22:03:43.223502 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223483    2566 scope.go:117] "RemoveContainer" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:03:43.223709 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.223692    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2\": container with ID starting with 85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2 not found: ID does not exist" containerID="85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"
Jul 16 22:03:43.223770 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223718    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2"} err="failed to get container status \"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2\": rpc error: code = NotFound desc = could not find container \"85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2\": container with ID starting with 85ddbaa26c5b851661f11ec749363f70ac0209cab5dd58db3f73e01058e79bc2 not found: ID does not exist"
Jul 16 22:03:43.223770 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223744    2566 scope.go:117] "RemoveContainer" containerID="21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"
Jul 16 22:03:43.223994 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.223976    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03\": container with ID starting with 21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03 not found: ID does not exist" containerID="21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"
Jul 16 22:03:43.224036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.223998    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03"} err="failed to get container status \"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03\": rpc error: code = NotFound desc = could not find container \"21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03\": container with ID starting with 21c3c2c90d8911db027df62e84deb8a20f6afdc1e3ca7db99d11de0e6f6a3f03 not found: ID does not exist"
Jul 16 22:03:43.224036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.224010    2566 scope.go:117] "RemoveContainer" containerID="7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"
Jul 16 22:03:43.224215 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:03:43.224198    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d\": container with ID starting with 7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d not found: ID does not exist" containerID="7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"
Jul 16 22:03:43.224265 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.224222    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d"} err="failed to get container status \"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d\": rpc error: code = NotFound desc = could not find container \"7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d\": container with ID starting with 7b2a819d6910cb6a3a5c302dfea89f6587ab56f09ab44f64b3a6cecc3e24676d not found: ID does not exist"
Jul 16 22:03:43.236621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236594    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.236713 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drwnq\" (UniqueName: \"kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.236713 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.236841 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hknct\" (UniqueName: \"kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.236841 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.236841 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.236990 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.236990 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.236990 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.236990 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.237183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.236988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home" (OuterVolumeSpecName: "home") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.237183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.237183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs\") pod \"3e403c51-0e76-4322-bb7d-6481a5c5d444\" (UID: \"3e403c51-0e76-4322-bb7d-6481a5c5d444\") "
Jul 16 22:03:43.237183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location\") pod \"517feab4-224b-4309-9286-c8cfe4236e75\" (UID: \"517feab4-224b-4309-9286-c8cfe4236e75\") "
Jul 16 22:03:43.237368 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237121    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.237368 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237324    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.237368 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237342    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.237368 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.237569 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.237569 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.237726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.237705    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.238217 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.238180    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.239661 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.239634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:03:43.239765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.239720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct" (OuterVolumeSpecName: "kube-api-access-hknct") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "kube-api-access-hknct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:03:43.239765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.239739    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq" (OuterVolumeSpecName: "kube-api-access-drwnq") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "kube-api-access-drwnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:03:43.239981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.239754    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm" (OuterVolumeSpecName: "dshm") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.240261 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.240243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "517feab4-224b-4309-9286-c8cfe4236e75" (UID: "517feab4-224b-4309-9286-c8cfe4236e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:03:43.292206 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.292160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e403c51-0e76-4322-bb7d-6481a5c5d444" (UID: "3e403c51-0e76-4322-bb7d-6481a5c5d444"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:03:43.338457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338385    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-tmp\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338415    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338423    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338431    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/517feab4-224b-4309-9286-c8cfe4236e75-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338469    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338481    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e403c51-0e76-4322-bb7d-6481a5c5d444-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338490    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/517feab4-224b-4309-9286-c8cfe4236e75-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338499    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338508    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drwnq\" (UniqueName: \"kubernetes.io/projected/3e403c51-0e76-4322-bb7d-6481a5c5d444-kube-api-access-drwnq\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338515    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknct\" (UniqueName: \"kubernetes.io/projected/517feab4-224b-4309-9286-c8cfe4236e75-kube-api-access-hknct\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.338688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.338523    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e403c51-0e76-4322-bb7d-6481a5c5d444-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:03:43.501501 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.501467    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:03:43.508738 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.508707    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7597885744-t6hgf"]
Jul 16 22:03:43.520451 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.520420    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:03:43.523989 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:43.523967    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-5bf6f948cxswm"]
Jul 16 22:03:44.319750 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:44.319716    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" path="/var/lib/kubelet/pods/3e403c51-0e76-4322-bb7d-6481a5c5d444/volumes"
Jul 16 22:03:44.320170 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:03:44.320155    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517feab4-224b-4309-9286-c8cfe4236e75" path="/var/lib/kubelet/pods/517feab4-224b-4309-9286-c8cfe4236e75/volumes"
Jul 16 22:04:09.641400 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641366    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641711    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641724    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="storage-initializer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641729    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="storage-initializer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641739    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641743    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641756    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641764    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="storage-initializer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="storage-initializer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641859    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641866    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="tokenizer"
Jul 16 22:04:09.641869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641875    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:04:09.642325 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.641880    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e403c51-0e76-4322-bb7d-6481a5c5d444" containerName="main"
Jul 16 22:04:09.646299 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.646283    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.650413 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.650393    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 16 22:04:09.650543 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.650395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-g9b65\""
Jul 16 22:04:09.656652 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.656627    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:04:09.738594 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.738537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.738780 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.738626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.738780 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.738659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccbc\" (UniqueName: \"kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.839423 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.839396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccbc\" (UniqueName: \"kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.839592 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.839448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.839658 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.839637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.839883 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.839861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.842245 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.842222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.849184 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.849158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccbc\" (UniqueName: \"kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") " pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:09.956297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:09.956259    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:10.109697 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:10.109661    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:04:10.112483 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:04:10.112455    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b885a7_6469_4262_87de_1ae5d24b7894.slice/crio-e498e5550d0db7370e231abe2b813b851d28cb3d827892b4ab1fcd5d7eb8513a WatchSource:0}: Error finding container e498e5550d0db7370e231abe2b813b851d28cb3d827892b4ab1fcd5d7eb8513a: Status 404 returned error can't find the container with id e498e5550d0db7370e231abe2b813b851d28cb3d827892b4ab1fcd5d7eb8513a
Jul 16 22:04:10.311632 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:10.311541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" event={"ID":"50b885a7-6469-4262-87de-1ae5d24b7894","Type":"ContainerStarted","Data":"c783981df3b34de897905669ae192b222fde198a0b532e02fb91df605ff87973"}
Jul 16 22:04:10.311632 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:10.311589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" event={"ID":"50b885a7-6469-4262-87de-1ae5d24b7894","Type":"ContainerStarted","Data":"e498e5550d0db7370e231abe2b813b851d28cb3d827892b4ab1fcd5d7eb8513a"}
Jul 16 22:04:10.311632 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:10.311618    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:10.334801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:10.334756    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" podStartSLOduration=1.334741089 podStartE2EDuration="1.334741089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:04:10.332937087 +0000 UTC m=+850.428448417" watchObservedRunningTime="2026-07-16 22:04:10.334741089 +0000 UTC m=+850.430252405"
Jul 16 22:04:41.318082 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:41.318049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:42.346366 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:42.346332    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:04:51.317980 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:51.317883    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:04:53.819847 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:53.819797    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:04:57.367715 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.367641    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" containerName="main" containerID="cri-o://c783981df3b34de897905669ae192b222fde198a0b532e02fb91df605ff87973" gracePeriod=45
Jul 16 22:04:57.536443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.536409    2566 generic.go:356] "Generic (PLEG): container finished" podID="50b885a7-6469-4262-87de-1ae5d24b7894" containerID="c783981df3b34de897905669ae192b222fde198a0b532e02fb91df605ff87973" exitCode=0
Jul 16 22:04:57.536595 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.536459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" event={"ID":"50b885a7-6469-4262-87de-1ae5d24b7894","Type":"ContainerDied","Data":"c783981df3b34de897905669ae192b222fde198a0b532e02fb91df605ff87973"}
Jul 16 22:04:57.626827 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.626756    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:57.666716 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.666686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds\") pod \"50b885a7-6469-4262-87de-1ae5d24b7894\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") "
Jul 16 22:04:57.666896 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.666725    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wccbc\" (UniqueName: \"kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc\") pod \"50b885a7-6469-4262-87de-1ae5d24b7894\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") "
Jul 16 22:04:57.666896 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.666754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs\") pod \"50b885a7-6469-4262-87de-1ae5d24b7894\" (UID: \"50b885a7-6469-4262-87de-1ae5d24b7894\") "
Jul 16 22:04:57.667053 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.667027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "50b885a7-6469-4262-87de-1ae5d24b7894" (UID: "50b885a7-6469-4262-87de-1ae5d24b7894"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:04:57.669074 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.669035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50b885a7-6469-4262-87de-1ae5d24b7894" (UID: "50b885a7-6469-4262-87de-1ae5d24b7894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:04:57.669188 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.669078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc" (OuterVolumeSpecName: "kube-api-access-wccbc") pod "50b885a7-6469-4262-87de-1ae5d24b7894" (UID: "50b885a7-6469-4262-87de-1ae5d24b7894"). InnerVolumeSpecName "kube-api-access-wccbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:04:57.767763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.767717    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccbc\" (UniqueName: \"kubernetes.io/projected/50b885a7-6469-4262-87de-1ae5d24b7894-kube-api-access-wccbc\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:04:57.767763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.767753    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b885a7-6469-4262-87de-1ae5d24b7894-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:04:57.767763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:57.767765    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/50b885a7-6469-4262-87de-1ae5d24b7894-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:04:58.542346 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:58.542314    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"
Jul 16 22:04:58.542346 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:58.542322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg" event={"ID":"50b885a7-6469-4262-87de-1ae5d24b7894","Type":"ContainerDied","Data":"e498e5550d0db7370e231abe2b813b851d28cb3d827892b4ab1fcd5d7eb8513a"}
Jul 16 22:04:58.542894 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:58.542386    2566 scope.go:117] "RemoveContainer" containerID="c783981df3b34de897905669ae192b222fde198a0b532e02fb91df605ff87973"
Jul 16 22:04:58.567800 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:58.567766    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:04:58.572867 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:04:58.572841    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schezhqlg"]
Jul 16 22:05:00.296878 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:00.296835    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:05:00.297287 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:00.297159    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:05:00.319997 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:00.319968    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" path="/var/lib/kubelet/pods/50b885a7-6469-4262-87de-1ae5d24b7894/volumes"
Jul 16 22:05:02.817360 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:02.817317    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:05:08.841437 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:08.841378    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="main" containerID="cri-o://647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7" gracePeriod=45
Jul 16 22:05:09.112424 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.112401    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:05:09.166107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.166064    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68bpp\" (UniqueName: \"kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp\") pod \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") "
Jul 16 22:05:09.166303 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.166160    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs\") pod \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") "
Jul 16 22:05:09.166303 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.166231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location\") pod \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") "
Jul 16 22:05:09.166303 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.166256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds\") pod \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\" (UID: \"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756\") "
Jul 16 22:05:09.166718 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.166677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" (UID: "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.167133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.167083    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" (UID: "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:05:09.168665 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.168635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp" (OuterVolumeSpecName: "kube-api-access-68bpp") pod "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" (UID: "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756"). InnerVolumeSpecName "kube-api-access-68bpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:05:09.168764 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.168677    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" (UID: "d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:05:09.241850 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.241800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:05:09.242181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242167    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="main"
Jul 16 22:05:09.242181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="main"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242193    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="storage-initializer"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242200    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="storage-initializer"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242210    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" containerName="main"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242215    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" containerName="main"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242224    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:05:09.242253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242231    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:05:09.242456 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242313    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="517feab4-224b-4309-9286-c8cfe4236e75" containerName="main"
Jul 16 22:05:09.242456 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242321    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b885a7-6469-4262-87de-1ae5d24b7894" containerName="main"
Jul 16 22:05:09.242456 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.242326    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerName="main"
Jul 16 22:05:09.245456 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.245434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.249479 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.249457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 16 22:05:09.267292 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.267260    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bpp\" (UniqueName: \"kubernetes.io/projected/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kube-api-access-68bpp\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.267292 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.267284    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.267292 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.267295    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.267527 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.267305    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:05:09.283728 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.283696    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:05:09.368226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.368226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.368439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7hsx\" (UniqueName: \"kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.368439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.368439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.368439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.368396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469165 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469165 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7hsx\" (UniqueName: \"kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469690 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469752 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.469752 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.469709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.471651 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.471627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.472107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.472087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.486031 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.486001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7hsx\" (UniqueName: \"kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.555843 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.555781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:09.595944 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.595908    2566 generic.go:356] "Generic (PLEG): container finished" podID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" containerID="647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7" exitCode=0
Jul 16 22:05:09.596108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.595971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerDied","Data":"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"}
Jul 16 22:05:09.596108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.596017    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"
Jul 16 22:05:09.596108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.596024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t" event={"ID":"d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756","Type":"ContainerDied","Data":"ece0318b5225735e6f3166df0fc903ed3068228ef3b192bad8fb5fe8d050f85c"}
Jul 16 22:05:09.596108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.596052    2566 scope.go:117] "RemoveContainer" containerID="647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"
Jul 16 22:05:09.607056 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.607036    2566 scope.go:117] "RemoveContainer" containerID="a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"
Jul 16 22:05:09.617855 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.617803    2566 scope.go:117] "RemoveContainer" containerID="647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"
Jul 16 22:05:09.618214 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:05:09.618192    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7\": container with ID starting with 647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7 not found: ID does not exist" containerID="647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"
Jul 16 22:05:09.618308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.618241    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7"} err="failed to get container status \"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7\": rpc error: code = NotFound desc = could not find container \"647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7\": container with ID starting with 647b63d4d4364bd5e2198243caa9b262b2d2e81ee2deb0c59cdc5b74beb9f4f7 not found: ID does not exist"
Jul 16 22:05:09.618308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.618261    2566 scope.go:117] "RemoveContainer" containerID="a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"
Jul 16 22:05:09.618573 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:05:09.618558    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b\": container with ID starting with a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b not found: ID does not exist" containerID="a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"
Jul 16 22:05:09.618640 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.618575    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b"} err="failed to get container status \"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b\": rpc error: code = NotFound desc = could not find container \"a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b\": container with ID starting with a2e93e4281b60242fdc87ed24a59dfb6e76cda8012184a8b0013f818c03bb07b not found: ID does not exist"
Jul 16 22:05:09.640097 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.640053    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:05:09.649705 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.649681    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-8444585c95-fwn2t"]
Jul 16 22:05:09.706633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.706601    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:05:09.711282 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.711262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.714879 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.714861    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-lnjsc\""
Jul 16 22:05:09.721455 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.721435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:05:09.724222 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:05:09.724198    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf629637_e202_4eb0_84e1_30f09df45771.slice/crio-a98638f48083995aa55b24bb8b1b7f3c03e5e5f0e5b58bef61ce4a9a5e142d4e WatchSource:0}: Error finding container a98638f48083995aa55b24bb8b1b7f3c03e5e5f0e5b58bef61ce4a9a5e142d4e: Status 404 returned error can't find the container with id a98638f48083995aa55b24bb8b1b7f3c03e5e5f0e5b58bef61ce4a9a5e142d4e
Jul 16 22:05:09.729238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.729217    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:05:09.772497 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.772472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxt7r\" (UniqueName: \"kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.772599 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.772563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.772599 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.772590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.873435 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.873340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxt7r\" (UniqueName: \"kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.874453 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.873446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.874453 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.873488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.874453 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.873920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.876447 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.876415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:09.887893 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:09.887870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxt7r\" (UniqueName: \"kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") " pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:10.020521 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.020481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:10.162585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.162512    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:05:10.165379 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:05:10.165353    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2780eec7_ffe0_497a_9b67_80cb3d99c130.slice/crio-030f52bacf8beb12639f536347178b9808a738de5e84c1d48bb701fbdc0996d4 WatchSource:0}: Error finding container 030f52bacf8beb12639f536347178b9808a738de5e84c1d48bb701fbdc0996d4: Status 404 returned error can't find the container with id 030f52bacf8beb12639f536347178b9808a738de5e84c1d48bb701fbdc0996d4
Jul 16 22:05:10.321550 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.321519    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756" path="/var/lib/kubelet/pods/d2ea3b40-ca2e-4ec2-9cb7-439a2ac1f756/volumes"
Jul 16 22:05:10.602107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.602065    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" event={"ID":"cf629637-e202-4eb0-84e1-30f09df45771","Type":"ContainerStarted","Data":"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"}
Jul 16 22:05:10.602282 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.602137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" event={"ID":"cf629637-e202-4eb0-84e1-30f09df45771","Type":"ContainerStarted","Data":"a98638f48083995aa55b24bb8b1b7f3c03e5e5f0e5b58bef61ce4a9a5e142d4e"}
Jul 16 22:05:10.604380 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.604351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" event={"ID":"2780eec7-ffe0-497a-9b67-80cb3d99c130","Type":"ContainerStarted","Data":"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"}
Jul 16 22:05:10.604505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.604400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" event={"ID":"2780eec7-ffe0-497a-9b67-80cb3d99c130","Type":"ContainerStarted","Data":"030f52bacf8beb12639f536347178b9808a738de5e84c1d48bb701fbdc0996d4"}
Jul 16 22:05:10.604505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.604475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:10.623701 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.623634    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" podStartSLOduration=1.6236122800000001 podStartE2EDuration="1.62361228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:10.622132416 +0000 UTC m=+910.717643734" watchObservedRunningTime="2026-07-16 22:05:10.62361228 +0000 UTC m=+910.719123598"
Jul 16 22:05:10.644084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:10.644002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" podStartSLOduration=1.643986436 podStartE2EDuration="1.643986436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:10.64267981 +0000 UTC m=+910.738191128" watchObservedRunningTime="2026-07-16 22:05:10.643986436 +0000 UTC m=+910.739497753"
Jul 16 22:05:11.556288 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:11.556251    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:19.556582 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:19.556546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:19.569049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:19.569021    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:19.664117 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:19.664091    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:05:41.615768 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:41.615740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:05:57.749034 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.748996    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:05:57.752583 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.752562    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.756582 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.756557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 16 22:05:57.756730 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.756583    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-747cz\""
Jul 16 22:05:57.775736 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.775708    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:05:57.800586 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.800550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.800586 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.800587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.800785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.800626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kngjm\" (UniqueName: \"kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.901437 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.901397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.901437 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.901435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.901655 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.901565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kngjm\" (UniqueName: \"kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.901874 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.901854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.904123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.904104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:57.910738 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:57.910705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kngjm\" (UniqueName: \"kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:58.063710 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:58.063622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:05:58.207581 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:58.207461    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:05:58.210470 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:05:58.210440    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8770c47d_04d6_45cd_8f2c_acc97e6101cf.slice/crio-ca205e680b2fd67ac50ac3e829af2d31b1b91516a51e629926bf7f0cf414e08b WatchSource:0}: Error finding container ca205e680b2fd67ac50ac3e829af2d31b1b91516a51e629926bf7f0cf414e08b: Status 404 returned error can't find the container with id ca205e680b2fd67ac50ac3e829af2d31b1b91516a51e629926bf7f0cf414e08b
Jul 16 22:05:58.835457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:58.835424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" event={"ID":"8770c47d-04d6-45cd-8f2c-acc97e6101cf","Type":"ContainerStarted","Data":"f060702a5e4dd3ad7beda581a5255c284cc462e5de6b7ef946b975609c7bc895"}
Jul 16 22:05:58.835870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:58.835471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" event={"ID":"8770c47d-04d6-45cd-8f2c-acc97e6101cf","Type":"ContainerStarted","Data":"ca205e680b2fd67ac50ac3e829af2d31b1b91516a51e629926bf7f0cf414e08b"}
Jul 16 22:05:58.835870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:05:58.835598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:06:29.841514 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:29.841426    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:06:29.859997 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:29.859944    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" podStartSLOduration=32.859927276 podStartE2EDuration="32.859927276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:05:58.866413535 +0000 UTC m=+958.961924852" watchObservedRunningTime="2026-07-16 22:06:29.859927276 +0000 UTC m=+989.955438594"
Jul 16 22:06:30.922307 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:30.922271    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:06:39.841616 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:39.841568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:06:45.944531 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:45.944487    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerName="main" containerID="cri-o://f060702a5e4dd3ad7beda581a5255c284cc462e5de6b7ef946b975609c7bc895" gracePeriod=45
Jul 16 22:06:46.062724 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.062696    2566 generic.go:356] "Generic (PLEG): container finished" podID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerID="f060702a5e4dd3ad7beda581a5255c284cc462e5de6b7ef946b975609c7bc895" exitCode=0
Jul 16 22:06:46.062805 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.062773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" event={"ID":"8770c47d-04d6-45cd-8f2c-acc97e6101cf","Type":"ContainerDied","Data":"f060702a5e4dd3ad7beda581a5255c284cc462e5de6b7ef946b975609c7bc895"}
Jul 16 22:06:46.202064 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.201993    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:06:46.255418 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.255383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs\") pod \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") "
Jul 16 22:06:46.255610 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.255481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds\") pod \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") "
Jul 16 22:06:46.255610 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.255509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kngjm\" (UniqueName: \"kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm\") pod \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\" (UID: \"8770c47d-04d6-45cd-8f2c-acc97e6101cf\") "
Jul 16 22:06:46.255764 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.255740    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8770c47d-04d6-45cd-8f2c-acc97e6101cf" (UID: "8770c47d-04d6-45cd-8f2c-acc97e6101cf"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:06:46.257662 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.257629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8770c47d-04d6-45cd-8f2c-acc97e6101cf" (UID: "8770c47d-04d6-45cd-8f2c-acc97e6101cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:06:46.257772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.257676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm" (OuterVolumeSpecName: "kube-api-access-kngjm") pod "8770c47d-04d6-45cd-8f2c-acc97e6101cf" (UID: "8770c47d-04d6-45cd-8f2c-acc97e6101cf"). InnerVolumeSpecName "kube-api-access-kngjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:06:46.357059 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.357023    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:06:46.357059 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.357052    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kngjm\" (UniqueName: \"kubernetes.io/projected/8770c47d-04d6-45cd-8f2c-acc97e6101cf-kube-api-access-kngjm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:06:46.357059 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:46.357065    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8770c47d-04d6-45cd-8f2c-acc97e6101cf-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:06:47.068633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:47.068598    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"
Jul 16 22:06:47.068633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:47.068620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv" event={"ID":"8770c47d-04d6-45cd-8f2c-acc97e6101cf","Type":"ContainerDied","Data":"ca205e680b2fd67ac50ac3e829af2d31b1b91516a51e629926bf7f0cf414e08b"}
Jul 16 22:06:47.069170 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:47.068672    2566 scope.go:117] "RemoveContainer" containerID="f060702a5e4dd3ad7beda581a5255c284cc462e5de6b7ef946b975609c7bc895"
Jul 16 22:06:47.094946 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:47.094915    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:06:47.101304 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:47.101277    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schecp7wv"]
Jul 16 22:06:48.319711 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:48.319678    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" path="/var/lib/kubelet/pods/8770c47d-04d6-45cd-8f2c-acc97e6101cf/volumes"
Jul 16 22:06:51.312924 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:51.312891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:06:51.324326 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:51.324295    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:06:51.615900 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:06:51.615793    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:07:01.615600 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:01.615565    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:07:06.334439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.334379    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main" containerID="cri-o://726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a" gracePeriod=45
Jul 16 22:07:06.347273 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.347220    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" podUID="cf629637-e202-4eb0-84e1-30f09df45771" containerName="main" containerID="cri-o://eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688" gracePeriod=45
Jul 16 22:07:06.631457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.631436    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:07:06.643648 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.643624    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:07:06.734449 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs\") pod \"2780eec7-ffe0-497a-9b67-80cb3d99c130\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") "
Jul 16 22:07:06.734449 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxt7r\" (UniqueName: \"kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r\") pod \"2780eec7-ffe0-497a-9b67-80cb3d99c130\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") "
Jul 16 22:07:06.734726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.734726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.734726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.734726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.734726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7hsx\" (UniqueName: \"kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.735041 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds\") pod \"2780eec7-ffe0-497a-9b67-80cb3d99c130\" (UID: \"2780eec7-ffe0-497a-9b67-80cb3d99c130\") "
Jul 16 22:07:06.735041 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home\") pod \"cf629637-e202-4eb0-84e1-30f09df45771\" (UID: \"cf629637-e202-4eb0-84e1-30f09df45771\") "
Jul 16 22:07:06.735041 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:06.735041 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.734857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:06.735263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.735052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "2780eec7-ffe0-497a-9b67-80cb3d99c130" (UID: "2780eec7-ffe0-497a-9b67-80cb3d99c130"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:06.735263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.735124    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.735263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.735146    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.735263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.735164    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/2780eec7-ffe0-497a-9b67-80cb3d99c130-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.735653 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.735420    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home" (OuterVolumeSpecName: "home") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:06.736972 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.736943    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx" (OuterVolumeSpecName: "kube-api-access-r7hsx") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "kube-api-access-r7hsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:07:06.736972 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.736964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r" (OuterVolumeSpecName: "kube-api-access-jxt7r") pod "2780eec7-ffe0-497a-9b67-80cb3d99c130" (UID: "2780eec7-ffe0-497a-9b67-80cb3d99c130"). InnerVolumeSpecName "kube-api-access-jxt7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:07:06.737196 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.737022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2780eec7-ffe0-497a-9b67-80cb3d99c130" (UID: "2780eec7-ffe0-497a-9b67-80cb3d99c130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:07:06.737327 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.737307    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm" (OuterVolumeSpecName: "dshm") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:07:06.737367 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.737309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf629637-e202-4eb0-84e1-30f09df45771" (UID: "cf629637-e202-4eb0-84e1-30f09df45771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:07:06.835909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835870    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf629637-e202-4eb0-84e1-30f09df45771-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.835909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835900    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7hsx\" (UniqueName: \"kubernetes.io/projected/cf629637-e202-4eb0-84e1-30f09df45771-kube-api-access-r7hsx\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.835909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835910    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.835909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835919    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2780eec7-ffe0-497a-9b67-80cb3d99c130-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.836177 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835928    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxt7r\" (UniqueName: \"kubernetes.io/projected/2780eec7-ffe0-497a-9b67-80cb3d99c130-kube-api-access-jxt7r\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:06.836177 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:06.835935    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf629637-e202-4eb0-84e1-30f09df45771-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:07:07.163114 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.163079    2566 generic.go:356] "Generic (PLEG): container finished" podID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerID="726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a" exitCode=0
Jul 16 22:07:07.163322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.163123    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"
Jul 16 22:07:07.163322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.163170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" event={"ID":"2780eec7-ffe0-497a-9b67-80cb3d99c130","Type":"ContainerDied","Data":"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"}
Jul 16 22:07:07.163322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.163225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9" event={"ID":"2780eec7-ffe0-497a-9b67-80cb3d99c130","Type":"ContainerDied","Data":"030f52bacf8beb12639f536347178b9808a738de5e84c1d48bb701fbdc0996d4"}
Jul 16 22:07:07.163322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.163255    2566 scope.go:117] "RemoveContainer" containerID="726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"
Jul 16 22:07:07.164684 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.164658    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf629637-e202-4eb0-84e1-30f09df45771" containerID="eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688" exitCode=0
Jul 16 22:07:07.164836 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.164693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" event={"ID":"cf629637-e202-4eb0-84e1-30f09df45771","Type":"ContainerDied","Data":"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"}
Jul 16 22:07:07.164836 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.164734    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"
Jul 16 22:07:07.164836 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.164744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9" event={"ID":"cf629637-e202-4eb0-84e1-30f09df45771","Type":"ContainerDied","Data":"a98638f48083995aa55b24bb8b1b7f3c03e5e5f0e5b58bef61ce4a9a5e142d4e"}
Jul 16 22:07:07.173514 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.173493    2566 scope.go:117] "RemoveContainer" containerID="726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"
Jul 16 22:07:07.173860 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:07:07.173841    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a\": container with ID starting with 726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a not found: ID does not exist" containerID="726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"
Jul 16 22:07:07.173935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.173869    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a"} err="failed to get container status \"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a\": rpc error: code = NotFound desc = could not find container \"726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a\": container with ID starting with 726be96a76e9e9d38dabd5202ebe197351a6bce095f77f50a46cd0f2c499d97a not found: ID does not exist"
Jul 16 22:07:07.173935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.173885    2566 scope.go:117] "RemoveContainer" containerID="eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"
Jul 16 22:07:07.182033 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.182013    2566 scope.go:117] "RemoveContainer" containerID="eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"
Jul 16 22:07:07.182308 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:07:07.182290    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688\": container with ID starting with eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688 not found: ID does not exist" containerID="eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"
Jul 16 22:07:07.182369 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.182315    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688"} err="failed to get container status \"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688\": rpc error: code = NotFound desc = could not find container \"eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688\": container with ID starting with eab62f434d5f6922daa862ea29b3bfd523d5395aa2f78452f326572132d9e688 not found: ID does not exist"
Jul 16 22:07:07.190036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.190016    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:07:07.196009 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.195971    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-scherkzd9"]
Jul 16 22:07:07.208219 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.208188    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:07:07.213051 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:07.213027    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4xbrg9"]
Jul 16 22:07:08.320081 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:08.320051    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" path="/var/lib/kubelet/pods/2780eec7-ffe0-497a-9b67-80cb3d99c130/volumes"
Jul 16 22:07:08.320469 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:08.320405    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf629637-e202-4eb0-84e1-30f09df45771" path="/var/lib/kubelet/pods/cf629637-e202-4eb0-84e1-30f09df45771/volumes"
Jul 16 22:07:21.601457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601424    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601741    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerName="main"
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerName="main"
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601771    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main"
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601777    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main"
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601789    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf629637-e202-4eb0-84e1-30f09df45771" containerName="main"
Jul 16 22:07:21.601810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601795    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf629637-e202-4eb0-84e1-30f09df45771" containerName="main"
Jul 16 22:07:21.602030 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601860    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf629637-e202-4eb0-84e1-30f09df45771" containerName="main"
Jul 16 22:07:21.602030 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601868    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2780eec7-ffe0-497a-9b67-80cb3d99c130" containerName="main"
Jul 16 22:07:21.602030 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.601874    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8770c47d-04d6-45cd-8f2c-acc97e6101cf" containerName="main"
Jul 16 22:07:21.606161 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.606142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.609087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.609063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:07:21.609087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.609081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:07:21.610102 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.610086    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:07:21.610227 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.610211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 16 22:07:21.620594 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.620568    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:07:21.658168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.658168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.658387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.658387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqh2\" (UniqueName: \"kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.658387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.658387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.658378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759603 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759611    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.759772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.759730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqh2\" (UniqueName: \"kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.760105 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.760081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.760167 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.760127    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.760167 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.760140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.761946 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.761926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.762187 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.762171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.768334 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.768313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqh2\" (UniqueName: \"kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.892865 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.892762    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:07:21.896711 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.896689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:21.899338 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.899317    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-b66m4\""
Jul 16 22:07:21.909758 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.909739    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:07:21.916076 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.916056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:21.961800 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.961750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7nx\" (UniqueName: \"kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:21.961996 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.961893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:21.961996 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:21.961928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.054630 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.054605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:07:22.056533 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:07:22.056503    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a1b7f8_7752_4782_bb2d_377d4f18c56c.slice/crio-762b572a5d26fbfd424d4234c2aaa156d81d51148f7bdd0c9156f4c284e57df2 WatchSource:0}: Error finding container 762b572a5d26fbfd424d4234c2aaa156d81d51148f7bdd0c9156f4c284e57df2: Status 404 returned error can't find the container with id 762b572a5d26fbfd424d4234c2aaa156d81d51148f7bdd0c9156f4c284e57df2
Jul 16 22:07:22.062743 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.062718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.062870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.062756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.062870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.062850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7nx\" (UniqueName: \"kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.063192 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.063163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.065371 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.065354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.071629 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.071604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7nx\" (UniqueName: \"kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") " pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.206716 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.206688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:22.238028 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.237997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" event={"ID":"c1a1b7f8-7752-4782-bb2d-377d4f18c56c","Type":"ContainerStarted","Data":"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"}
Jul 16 22:07:22.238153 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.238076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" event={"ID":"c1a1b7f8-7752-4782-bb2d-377d4f18c56c","Type":"ContainerStarted","Data":"762b572a5d26fbfd424d4234c2aaa156d81d51148f7bdd0c9156f4c284e57df2"}
Jul 16 22:07:22.266874 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.266808    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" podStartSLOduration=1.2667914169999999 podStartE2EDuration="1.266791417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:07:22.265335229 +0000 UTC m=+1042.360846544" watchObservedRunningTime="2026-07-16 22:07:22.266791417 +0000 UTC m=+1042.362302733"
Jul 16 22:07:22.342582 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.342556    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:07:22.343563 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:07:22.343525    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3633686_7335_4876_a684_80652f40b31b.slice/crio-0c38bd0278cfbd7d6d7745daed823b85f4b6795f183f5c0d5d3e78b8fef3265b WatchSource:0}: Error finding container 0c38bd0278cfbd7d6d7745daed823b85f4b6795f183f5c0d5d3e78b8fef3265b: Status 404 returned error can't find the container with id 0c38bd0278cfbd7d6d7745daed823b85f4b6795f183f5c0d5d3e78b8fef3265b
Jul 16 22:07:22.916928 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:22.916889    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:23.245051 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:23.245015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" event={"ID":"b3633686-7335-4876-a684-80652f40b31b","Type":"ContainerStarted","Data":"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"}
Jul 16 22:07:23.245272 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:23.245069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" event={"ID":"b3633686-7335-4876-a684-80652f40b31b","Type":"ContainerStarted","Data":"0c38bd0278cfbd7d6d7745daed823b85f4b6795f183f5c0d5d3e78b8fef3265b"}
Jul 16 22:07:23.245272 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:23.245093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:07:23.272419 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:23.272366    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" podStartSLOduration=2.272351598 podStartE2EDuration="2.272351598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:07:23.270698559 +0000 UTC m=+1043.366209876" watchObservedRunningTime="2026-07-16 22:07:23.272351598 +0000 UTC m=+1043.367862915"
Jul 16 22:07:31.916646 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:31.916609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:31.929107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:31.929081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:32.299026 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:32.298998    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:07:54.251452 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:07:54.251372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:08:06.930153 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.930111    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:08:06.935490 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.935465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:06.940219 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.940193    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-xctnl\""
Jul 16 22:08:06.940375 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.940197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 16 22:08:06.950175 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.950149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:08:06.964701 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.964665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:06.964882 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.964811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:06.964940 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:06.964923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9xk\" (UniqueName: \"kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.065755 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.065721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.065959 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.065780    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9xk\" (UniqueName: \"kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.065959 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.065806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.066162 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.066143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.068364 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.068335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.077884 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.077860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9xk\" (UniqueName: \"kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.245183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.245140    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:07.392346 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.392316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:08:07.395440 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:08:07.395413    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadc9f34_6a15_41eb_a4bb_5d4e7b848c9f.slice/crio-45ce0aeb64913ea046dbb713920dff8be30c46993a9b468933e357e9d635cb81 WatchSource:0}: Error finding container 45ce0aeb64913ea046dbb713920dff8be30c46993a9b468933e357e9d635cb81: Status 404 returned error can't find the container with id 45ce0aeb64913ea046dbb713920dff8be30c46993a9b468933e357e9d635cb81
Jul 16 22:08:07.461129 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:07.461087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" event={"ID":"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f","Type":"ContainerStarted","Data":"45ce0aeb64913ea046dbb713920dff8be30c46993a9b468933e357e9d635cb81"}
Jul 16 22:08:08.468246 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:08.468211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" event={"ID":"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f","Type":"ContainerStarted","Data":"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"}
Jul 16 22:08:08.468642 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:08.468296    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:08:08.489552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:08.489494    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" podStartSLOduration=2.489478483 podStartE2EDuration="2.489478483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:08:08.487284983 +0000 UTC m=+1088.582796337" watchObservedRunningTime="2026-07-16 22:08:08.489478483 +0000 UTC m=+1088.584989800"
Jul 16 22:08:39.475879 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:08:39.475850    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:09:15.021952 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:15.021901    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:09:19.476375 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:19.476337    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:09:29.475500 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:29.475413    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:09:30.043338 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.043273    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main" containerID="cri-o://e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5" gracePeriod=45
Jul 16 22:09:30.296045 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.295982    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:09:30.383508 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.383469    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw9xk\" (UniqueName: \"kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk\") pod \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") "
Jul 16 22:09:30.383684 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.383534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs\") pod \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") "
Jul 16 22:09:30.383684 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.383560    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds\") pod \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\" (UID: \"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f\") "
Jul 16 22:09:30.383876 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.383849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" (UID: "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:30.384111 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.384089    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:30.385878 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.385852    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk" (OuterVolumeSpecName: "kube-api-access-bw9xk") pod "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" (UID: "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f"). InnerVolumeSpecName "kube-api-access-bw9xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:09:30.385974 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.385870    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" (UID: "eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:09:30.485244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.485205    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9xk\" (UniqueName: \"kubernetes.io/projected/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-kube-api-access-bw9xk\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:30.485244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.485240    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:30.859967 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.859926    2566 generic.go:356] "Generic (PLEG): container finished" podID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerID="e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5" exitCode=0
Jul 16 22:09:30.860179 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.859975    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"
Jul 16 22:09:30.860179 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.859984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" event={"ID":"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f","Type":"ContainerDied","Data":"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"}
Jul 16 22:09:30.860179 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.860024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl" event={"ID":"eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f","Type":"ContainerDied","Data":"45ce0aeb64913ea046dbb713920dff8be30c46993a9b468933e357e9d635cb81"}
Jul 16 22:09:30.860179 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.860042    2566 scope.go:117] "RemoveContainer" containerID="e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"
Jul 16 22:09:30.869253 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.869235    2566 scope.go:117] "RemoveContainer" containerID="e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"
Jul 16 22:09:30.869545 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:09:30.869525    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5\": container with ID starting with e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5 not found: ID does not exist" containerID="e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"
Jul 16 22:09:30.869643 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.869552    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5"} err="failed to get container status \"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5\": rpc error: code = NotFound desc = could not find container \"e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5\": container with ID starting with e624d24c776fb70693fad27fb0fe17085a9e57040b0f2434030fc90df9ae79b5 not found: ID does not exist"
Jul 16 22:09:30.882385 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.882359    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:09:30.886989 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:30.886962    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherdpsl"]
Jul 16 22:09:32.320085 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:32.320052    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" path="/var/lib/kubelet/pods/eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f/volumes"
Jul 16 22:09:34.937676 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:34.937638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:09:34.951743 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:34.951709    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:09:44.251756 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:44.251717    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" podUID="b3633686-7335-4876-a684-80652f40b31b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:09:49.960859 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:49.960803    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" podUID="b3633686-7335-4876-a684-80652f40b31b" containerName="main" containerID="cri-o://fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35" gracePeriod=45
Jul 16 22:09:49.972323 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:49.972290    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" podUID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" containerName="main" containerID="cri-o://2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3" gracePeriod=45
Jul 16 22:09:50.244002 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.243980    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:09:50.246609 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.246594    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:09:50.362460 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362421    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh7nx\" (UniqueName: \"kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx\") pod \"b3633686-7335-4876-a684-80652f40b31b\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") "
Jul 16 22:09:50.362460 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362464    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs\") pod \"b3633686-7335-4876-a684-80652f40b31b\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362592    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds\") pod \"b3633686-7335-4876-a684-80652f40b31b\" (UID: \"b3633686-7335-4876-a684-80652f40b31b\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.362703 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkqh2\" (UniqueName: \"kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2\") pod \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\" (UID: \"c1a1b7f8-7752-4782-bb2d-377d4f18c56c\") "
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362867    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b3633686-7335-4876-a684-80652f40b31b" (UID: "b3633686-7335-4876-a684-80652f40b31b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home" (OuterVolumeSpecName: "home") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.362996    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.363029    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b3633686-7335-4876-a684-80652f40b31b-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.363080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.363056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:50.365068 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.365043    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:09:50.365190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.365150    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm" (OuterVolumeSpecName: "dshm") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:09:50.365249 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.365195    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2" (OuterVolumeSpecName: "kube-api-access-dkqh2") pod "c1a1b7f8-7752-4782-bb2d-377d4f18c56c" (UID: "c1a1b7f8-7752-4782-bb2d-377d4f18c56c"). InnerVolumeSpecName "kube-api-access-dkqh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:09:50.365292 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.365262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx" (OuterVolumeSpecName: "kube-api-access-xh7nx") pod "b3633686-7335-4876-a684-80652f40b31b" (UID: "b3633686-7335-4876-a684-80652f40b31b"). InnerVolumeSpecName "kube-api-access-xh7nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:09:50.365326 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.365299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3633686-7335-4876-a684-80652f40b31b" (UID: "b3633686-7335-4876-a684-80652f40b31b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:09:50.464075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464049    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7nx\" (UniqueName: \"kubernetes.io/projected/b3633686-7335-4876-a684-80652f40b31b-kube-api-access-xh7nx\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464070    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3633686-7335-4876-a684-80652f40b31b-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464079    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464088    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464096    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464104    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.464268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.464111    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqh2\" (UniqueName: \"kubernetes.io/projected/c1a1b7f8-7752-4782-bb2d-377d4f18c56c-kube-api-access-dkqh2\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:09:50.954083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.954045    2566 generic.go:356] "Generic (PLEG): container finished" podID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" containerID="2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3" exitCode=0
Jul 16 22:09:50.954283 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.954104    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"
Jul 16 22:09:50.954283 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.954119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" event={"ID":"c1a1b7f8-7752-4782-bb2d-377d4f18c56c","Type":"ContainerDied","Data":"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"}
Jul 16 22:09:50.954283 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.954164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk" event={"ID":"c1a1b7f8-7752-4782-bb2d-377d4f18c56c","Type":"ContainerDied","Data":"762b572a5d26fbfd424d4234c2aaa156d81d51148f7bdd0c9156f4c284e57df2"}
Jul 16 22:09:50.954283 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.954185    2566 scope.go:117] "RemoveContainer" containerID="2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"
Jul 16 22:09:50.955386 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.955365    2566 generic.go:356] "Generic (PLEG): container finished" podID="b3633686-7335-4876-a684-80652f40b31b" containerID="fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35" exitCode=0
Jul 16 22:09:50.955477 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.955426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"
Jul 16 22:09:50.955477 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.955443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" event={"ID":"b3633686-7335-4876-a684-80652f40b31b","Type":"ContainerDied","Data":"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"}
Jul 16 22:09:50.955477 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.955472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj" event={"ID":"b3633686-7335-4876-a684-80652f40b31b","Type":"ContainerDied","Data":"0c38bd0278cfbd7d6d7745daed823b85f4b6795f183f5c0d5d3e78b8fef3265b"}
Jul 16 22:09:50.964149 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.963956    2566 scope.go:117] "RemoveContainer" containerID="2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"
Jul 16 22:09:50.964415 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:09:50.964261    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3\": container with ID starting with 2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3 not found: ID does not exist" containerID="2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"
Jul 16 22:09:50.964415 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.964288    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3"} err="failed to get container status \"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3\": rpc error: code = NotFound desc = could not find container \"2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3\": container with ID starting with 2072b329d0281150b28205f971fe3257d0a8c2c30fffe4e6dd039a32f6ad15b3 not found: ID does not exist"
Jul 16 22:09:50.964415 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.964304    2566 scope.go:117] "RemoveContainer" containerID="fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"
Jul 16 22:09:50.972444 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.972428    2566 scope.go:117] "RemoveContainer" containerID="fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"
Jul 16 22:09:50.972708 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:09:50.972691    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35\": container with ID starting with fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35 not found: ID does not exist" containerID="fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"
Jul 16 22:09:50.972757 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.972715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35"} err="failed to get container status \"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35\": rpc error: code = NotFound desc = could not find container \"fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35\": container with ID starting with fbc8f58e2cb61f405f80810240e73fd366d15dd759c3e0dc4c7b054e82bdde35 not found: ID does not exist"
Jul 16 22:09:50.976566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.976545    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:09:50.981741 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.981723    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-rrpmk"]
Jul 16 22:09:50.992913 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.992883    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:09:50.994659 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:50.994641    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche7vqpj"]
Jul 16 22:09:52.319090 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:52.319057    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3633686-7335-4876-a684-80652f40b31b" path="/var/lib/kubelet/pods/b3633686-7335-4876-a684-80652f40b31b/volumes"
Jul 16 22:09:52.319457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:09:52.319416    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" path="/var/lib/kubelet/pods/c1a1b7f8-7752-4782-bb2d-377d4f18c56c/volumes"
Jul 16 22:10:00.324263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:00.324233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:10:00.325198 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:00.325163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:10:02.514930 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.514897    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515219    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3633686-7335-4876-a684-80652f40b31b" containerName="main"
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515230    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3633686-7335-4876-a684-80652f40b31b" containerName="main"
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515247    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" containerName="main"
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515252    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" containerName="main"
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515263    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main"
Jul 16 22:10:02.515316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515268    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main"
Jul 16 22:10:02.515622 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515326    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a1b7f8-7752-4782-bb2d-377d4f18c56c" containerName="main"
Jul 16 22:10:02.515622 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515334    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3633686-7335-4876-a684-80652f40b31b" containerName="main"
Jul 16 22:10:02.515622 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.515341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eadc9f34-6a15-41eb-a4bb-5d4e7b848c9f" containerName="main"
Jul 16 22:10:02.520069 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.520047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.522744 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.522722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 16 22:10:02.523808 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.523789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:10:02.523808 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.523858    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:10:02.524032 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.523890    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:10:02.524032 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.523803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-rj5pp\""
Jul 16 22:10:02.524032 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.523800    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:10:02.527168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.527147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:10:02.573254 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.573211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.573254 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.573251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.573468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.573288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.573468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.573316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvq4k\" (UniqueName: \"kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.673975 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.673941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.673975 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.673988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvq4k\" (UniqueName: \"kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.674197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.674031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.674197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.674048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.674473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.674449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.674511 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.674466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.676673 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.676651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.682323 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.682291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvq4k\" (UniqueName: \"kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.831167 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.831073    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:02.980752 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:02.980714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:10:02.985517 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:10:02.985486    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f21a94_17bf_4d08_b0b3_60d38c826d46.slice/crio-f38830b3e14a595bd0ebcb389e7b80e51bab03f0da229b5d346513d9311aa191 WatchSource:0}: Error finding container f38830b3e14a595bd0ebcb389e7b80e51bab03f0da229b5d346513d9311aa191: Status 404 returned error can't find the container with id f38830b3e14a595bd0ebcb389e7b80e51bab03f0da229b5d346513d9311aa191
Jul 16 22:10:03.013157 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:03.013115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" event={"ID":"c9f21a94-17bf-4d08-b0b3-60d38c826d46","Type":"ContainerStarted","Data":"f38830b3e14a595bd0ebcb389e7b80e51bab03f0da229b5d346513d9311aa191"}
Jul 16 22:10:04.019151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:04.019118    2566 generic.go:356] "Generic (PLEG): container finished" podID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerID="b7aa016a01c13dcf7b8c32bbcf626b72a6df5cc0ec85834bc2d43e77ca22fde9" exitCode=0
Jul 16 22:10:04.019151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:04.019155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" event={"ID":"c9f21a94-17bf-4d08-b0b3-60d38c826d46","Type":"ContainerDied","Data":"b7aa016a01c13dcf7b8c32bbcf626b72a6df5cc0ec85834bc2d43e77ca22fde9"}
Jul 16 22:10:05.025452 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:05.025413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" event={"ID":"c9f21a94-17bf-4d08-b0b3-60d38c826d46","Type":"ContainerStarted","Data":"aa63fd603863320973a4f31842b84f8feae11d6c9ddec911370fc86389b0dbc0"}
Jul 16 22:10:05.025884 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:05.025548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:10:05.045589 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:05.045531    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" podStartSLOduration=2.304547195 podStartE2EDuration="3.045514339s" totalImagesPullingTime="0s" totalInitContainerRuntime="740.967144ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:10:05.0443137 +0000 UTC m=+1205.139825008" watchObservedRunningTime="2026-07-16 22:10:05.045514339 +0000 UTC m=+1205.141025656"
Jul 16 22:10:36.032398 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:10:36.032370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:12:02.511582 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.511543    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:12:02.515157 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.515130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.519541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.519510    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 16 22:12:02.520568 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.520549    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-dkcv7\""
Jul 16 22:12:02.530810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.530772    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:12:02.605796 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.605754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.605796 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.605801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.606044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.605882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.606044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.605912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmt7\" (UniqueName: \"kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.706872 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.706802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.706872 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.706870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmt7\" (UniqueName: \"kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.707133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.706922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.707133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.706957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.707299 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.707273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.707354 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.707307    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.709533 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.709505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.715979 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.715954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmt7\" (UniqueName: \"kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.826350 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.826248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:02.972310 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:02.972283    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:12:02.974454 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:12:02.974420    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0e55ef_d33b_4cf6_abf8_887db4179b0e.slice/crio-8ce187682e12fd21d154f12dfe3752a59e13c8697e9d3d4f6d460155a7ecf593 WatchSource:0}: Error finding container 8ce187682e12fd21d154f12dfe3752a59e13c8697e9d3d4f6d460155a7ecf593: Status 404 returned error can't find the container with id 8ce187682e12fd21d154f12dfe3752a59e13c8697e9d3d4f6d460155a7ecf593
Jul 16 22:12:03.572547 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:03.572509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerStarted","Data":"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"}
Jul 16 22:12:03.573066 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:03.572569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerStarted","Data":"8ce187682e12fd21d154f12dfe3752a59e13c8697e9d3d4f6d460155a7ecf593"}
Jul 16 22:12:04.579572 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:04.579531    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerID="81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613" exitCode=0
Jul 16 22:12:04.580010 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:04.579634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerDied","Data":"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"}
Jul 16 22:12:05.434789 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:05.434756    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:12:05.586742 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:05.586707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerStarted","Data":"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"}
Jul 16 22:12:05.587149 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:05.586854    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:05.621337 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:05.621276    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" podStartSLOduration=2.870564999 podStartE2EDuration="3.621260173s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.695174ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:05.619937924 +0000 UTC m=+1325.715449261" watchObservedRunningTime="2026-07-16 22:12:05.621260173 +0000 UTC m=+1325.716771487"
Jul 16 22:12:06.032768 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:06.032729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:12:16.032871 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:16.032808    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:12:20.455039 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.454998    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main" containerID="cri-o://aa63fd603863320973a4f31842b84f8feae11d6c9ddec911370fc86389b0dbc0" gracePeriod=45
Jul 16 22:12:20.659359 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.659319    2566 generic.go:356] "Generic (PLEG): container finished" podID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerID="aa63fd603863320973a4f31842b84f8feae11d6c9ddec911370fc86389b0dbc0" exitCode=0
Jul 16 22:12:20.659504 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.659396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" event={"ID":"c9f21a94-17bf-4d08-b0b3-60d38c826d46","Type":"ContainerDied","Data":"aa63fd603863320973a4f31842b84f8feae11d6c9ddec911370fc86389b0dbc0"}
Jul 16 22:12:20.710630 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.710569    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:12:20.855590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.855560    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location\") pod \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") "
Jul 16 22:12:20.855748 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.855601    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds\") pod \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") "
Jul 16 22:12:20.855748 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.855628    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs\") pod \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") "
Jul 16 22:12:20.855748 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.855652    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvq4k\" (UniqueName: \"kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k\") pod \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\" (UID: \"c9f21a94-17bf-4d08-b0b3-60d38c826d46\") "
Jul 16 22:12:20.856072 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.856033    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c9f21a94-17bf-4d08-b0b3-60d38c826d46" (UID: "c9f21a94-17bf-4d08-b0b3-60d38c826d46"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:20.856359 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.856338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c9f21a94-17bf-4d08-b0b3-60d38c826d46" (UID: "c9f21a94-17bf-4d08-b0b3-60d38c826d46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:12:20.857915 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.857888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9f21a94-17bf-4d08-b0b3-60d38c826d46" (UID: "c9f21a94-17bf-4d08-b0b3-60d38c826d46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:20.858050 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.858027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k" (OuterVolumeSpecName: "kube-api-access-kvq4k") pod "c9f21a94-17bf-4d08-b0b3-60d38c826d46" (UID: "c9f21a94-17bf-4d08-b0b3-60d38c826d46"). InnerVolumeSpecName "kube-api-access-kvq4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:20.956474 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.956439    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:12:20.956474 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.956469    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:12:20.956474 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.956481    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9f21a94-17bf-4d08-b0b3-60d38c826d46-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:12:20.956720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:20.956489    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvq4k\" (UniqueName: \"kubernetes.io/projected/c9f21a94-17bf-4d08-b0b3-60d38c826d46-kube-api-access-kvq4k\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:12:21.666118 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.666091    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"
Jul 16 22:12:21.666118 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.666107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r" event={"ID":"c9f21a94-17bf-4d08-b0b3-60d38c826d46","Type":"ContainerDied","Data":"f38830b3e14a595bd0ebcb389e7b80e51bab03f0da229b5d346513d9311aa191"}
Jul 16 22:12:21.666607 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.666162    2566 scope.go:117] "RemoveContainer" containerID="aa63fd603863320973a4f31842b84f8feae11d6c9ddec911370fc86389b0dbc0"
Jul 16 22:12:21.675716 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.675698    2566 scope.go:117] "RemoveContainer" containerID="b7aa016a01c13dcf7b8c32bbcf626b72a6df5cc0ec85834bc2d43e77ca22fde9"
Jul 16 22:12:21.693575 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.693543    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:12:21.696827 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:21.696792    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2l22r"]
Jul 16 22:12:22.319345 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:22.319311    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" path="/var/lib/kubelet/pods/c9f21a94-17bf-4d08-b0b3-60d38c826d46/volumes"
Jul 16 22:12:36.592954 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.592925    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:12:36.671590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.671550    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:12:36.671977 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.671958    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="storage-initializer"
Jul 16 22:12:36.672068 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.671978    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="storage-initializer"
Jul 16 22:12:36.672068 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.671992    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main"
Jul 16 22:12:36.672068 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.672000    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main"
Jul 16 22:12:36.672223 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.672121    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9f21a94-17bf-4d08-b0b3-60d38c826d46" containerName="main"
Jul 16 22:12:36.675436 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.675416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.679522 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.679497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 16 22:12:36.679647 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.679547    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-rntkp\""
Jul 16 22:12:36.692207 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.692184    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:12:36.795933 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.795902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.796131 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.795948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sj6\" (UniqueName: \"kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.796131 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.796001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.796131 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.796039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.896954 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.896863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.896954 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.896916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sj6\" (UniqueName: \"kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.896954 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.896944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.897241 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.896984    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.897297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.897266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.897297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.897286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.899560 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.899538    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.908314 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.908286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sj6\" (UniqueName: \"kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:36.985866 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:36.985802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:37.127832 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:37.127776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:12:37.129140 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:12:37.129099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f4ff3_456e_4c21_8a07_f1d295228b9a.slice/crio-576e7609a1ae219e7d0c90d0fdfa85bfde6f7a7c97f7a7199223c3d294feb0d4 WatchSource:0}: Error finding container 576e7609a1ae219e7d0c90d0fdfa85bfde6f7a7c97f7a7199223c3d294feb0d4: Status 404 returned error can't find the container with id 576e7609a1ae219e7d0c90d0fdfa85bfde6f7a7c97f7a7199223c3d294feb0d4
Jul 16 22:12:37.742858 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:37.742791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerStarted","Data":"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"}
Jul 16 22:12:37.743223 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:37.742883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerStarted","Data":"576e7609a1ae219e7d0c90d0fdfa85bfde6f7a7c97f7a7199223c3d294feb0d4"}
Jul 16 22:12:38.749708 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:38.749668    2566 generic.go:356] "Generic (PLEG): container finished" podID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerID="3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1" exitCode=0
Jul 16 22:12:38.750139 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:38.749724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerDied","Data":"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"}
Jul 16 22:12:39.755879 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:39.755836    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerStarted","Data":"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"}
Jul 16 22:12:39.756299 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:39.755959    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:12:39.781491 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:12:39.781441    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" podStartSLOduration=3.035177058 podStartE2EDuration="3.78142061s" totalImagesPullingTime="0s" totalInitContainerRuntime="746.243552ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:39.779224661 +0000 UTC m=+1359.874735981" watchObservedRunningTime="2026-07-16 22:12:39.78142061 +0000 UTC m=+1359.876931929"
Jul 16 22:13:10.763367 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:13:10.763338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:14:07.963052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:07.962958    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:14:10.466901 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:10.466866    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:14:10.763055 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:10.762964    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:14:16.593238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:16.593199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:14:20.764051 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:20.764000    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:14:22.984446 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:22.984382    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="main" containerID="cri-o://b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e" gracePeriod=45
Jul 16 22:14:23.240099 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.240035    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:14:23.247393 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.247360    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerID="b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e" exitCode=0
Jul 16 22:14:23.247541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.247415    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"
Jul 16 22:14:23.247541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.247424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerDied","Data":"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"}
Jul 16 22:14:23.247541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.247474    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr" event={"ID":"1c0e55ef-d33b-4cf6-abf8-887db4179b0e","Type":"ContainerDied","Data":"8ce187682e12fd21d154f12dfe3752a59e13c8697e9d3d4f6d460155a7ecf593"}
Jul 16 22:14:23.247541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.247502    2566 scope.go:117] "RemoveContainer" containerID="b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"
Jul 16 22:14:23.255902 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.255884    2566 scope.go:117] "RemoveContainer" containerID="81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"
Jul 16 22:14:23.264152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.264123    2566 scope.go:117] "RemoveContainer" containerID="b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"
Jul 16 22:14:23.264482 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:14:23.264458    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e\": container with ID starting with b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e not found: ID does not exist" containerID="b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"
Jul 16 22:14:23.264574 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.264494    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e"} err="failed to get container status \"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e\": rpc error: code = NotFound desc = could not find container \"b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e\": container with ID starting with b8a35b957e04618b08085b751c5aac7c25b8c8de6dc07c10cfbd06bd68c21b0e not found: ID does not exist"
Jul 16 22:14:23.264574 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.264519    2566 scope.go:117] "RemoveContainer" containerID="81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"
Jul 16 22:14:23.264859 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:14:23.264809    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613\": container with ID starting with 81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613 not found: ID does not exist" containerID="81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"
Jul 16 22:14:23.264934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.264867    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613"} err="failed to get container status \"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613\": rpc error: code = NotFound desc = could not find container \"81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613\": container with ID starting with 81d21ac2ea94006d3613c75063c70b8c412883ae85d6a4b182d905bed4b00613 not found: ID does not exist"
Jul 16 22:14:23.430302 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430270    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location\") pod \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") "
Jul 16 22:14:23.430498 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430337    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds\") pod \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") "
Jul 16 22:14:23.430498 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430374    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdmt7\" (UniqueName: \"kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7\") pod \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") "
Jul 16 22:14:23.430498 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs\") pod \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\" (UID: \"1c0e55ef-d33b-4cf6-abf8-887db4179b0e\") "
Jul 16 22:14:23.430691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430565    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1c0e55ef-d33b-4cf6-abf8-887db4179b0e" (UID: "1c0e55ef-d33b-4cf6-abf8-887db4179b0e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.430801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.430779    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.431095 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.431072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c0e55ef-d33b-4cf6-abf8-887db4179b0e" (UID: "1c0e55ef-d33b-4cf6-abf8-887db4179b0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:23.433134 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.433080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c0e55ef-d33b-4cf6-abf8-887db4179b0e" (UID: "1c0e55ef-d33b-4cf6-abf8-887db4179b0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:14:23.433550 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.433530    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7" (OuterVolumeSpecName: "kube-api-access-zdmt7") pod "1c0e55ef-d33b-4cf6-abf8-887db4179b0e" (UID: "1c0e55ef-d33b-4cf6-abf8-887db4179b0e"). InnerVolumeSpecName "kube-api-access-zdmt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:23.532136 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.532048    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.532136 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.532082    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.532136 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.532093    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdmt7\" (UniqueName: \"kubernetes.io/projected/1c0e55ef-d33b-4cf6-abf8-887db4179b0e-kube-api-access-zdmt7\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:23.571073 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.571035    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:14:23.577007 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:23.576982    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexlpkr"]
Jul 16 22:14:24.320139 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:24.320111    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" path="/var/lib/kubelet/pods/1c0e55ef-d33b-4cf6-abf8-887db4179b0e/volumes"
Jul 16 22:14:25.487680 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.487629    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main" containerID="cri-o://f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba" gracePeriod=45
Jul 16 22:14:25.743311 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.743246    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:14:25.850922 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.850885    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87sj6\" (UniqueName: \"kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6\") pod \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") "
Jul 16 22:14:25.851115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.850951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds\") pod \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") "
Jul 16 22:14:25.851115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.850990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs\") pod \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") "
Jul 16 22:14:25.851115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.851017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location\") pod \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\" (UID: \"c52f4ff3-456e-4c21-8a07-f1d295228b9a\") "
Jul 16 22:14:25.851257 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.851229    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c52f4ff3-456e-4c21-8a07-f1d295228b9a" (UID: "c52f4ff3-456e-4c21-8a07-f1d295228b9a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:25.851669 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.851643    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c52f4ff3-456e-4c21-8a07-f1d295228b9a" (UID: "c52f4ff3-456e-4c21-8a07-f1d295228b9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:14:25.853227 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.853183    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6" (OuterVolumeSpecName: "kube-api-access-87sj6") pod "c52f4ff3-456e-4c21-8a07-f1d295228b9a" (UID: "c52f4ff3-456e-4c21-8a07-f1d295228b9a"). InnerVolumeSpecName "kube-api-access-87sj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:25.853319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.853252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c52f4ff3-456e-4c21-8a07-f1d295228b9a" (UID: "c52f4ff3-456e-4c21-8a07-f1d295228b9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:14:25.951621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.951586    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.951621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.951622    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.951858 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.951636    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87sj6\" (UniqueName: \"kubernetes.io/projected/c52f4ff3-456e-4c21-8a07-f1d295228b9a-kube-api-access-87sj6\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:25.951858 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:25.951645    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c52f4ff3-456e-4c21-8a07-f1d295228b9a-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:26.262921 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.262888    2566 generic.go:356] "Generic (PLEG): container finished" podID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerID="f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba" exitCode=0
Jul 16 22:14:26.263092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.262940    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"
Jul 16 22:14:26.263092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.262948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerDied","Data":"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"}
Jul 16 22:14:26.263092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.262989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j" event={"ID":"c52f4ff3-456e-4c21-8a07-f1d295228b9a","Type":"ContainerDied","Data":"576e7609a1ae219e7d0c90d0fdfa85bfde6f7a7c97f7a7199223c3d294feb0d4"}
Jul 16 22:14:26.263092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.263007    2566 scope.go:117] "RemoveContainer" containerID="f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"
Jul 16 22:14:26.272127 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.272108    2566 scope.go:117] "RemoveContainer" containerID="3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"
Jul 16 22:14:26.280084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.280059    2566 scope.go:117] "RemoveContainer" containerID="f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"
Jul 16 22:14:26.280365 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:14:26.280346    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba\": container with ID starting with f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba not found: ID does not exist" containerID="f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"
Jul 16 22:14:26.280439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.280378    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba"} err="failed to get container status \"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba\": rpc error: code = NotFound desc = could not find container \"f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba\": container with ID starting with f609ae5987a7ab516449ad1531ed4f043c4a39cd7332e1d626e7d147b74fe1ba not found: ID does not exist"
Jul 16 22:14:26.280439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.280402    2566 scope.go:117] "RemoveContainer" containerID="3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"
Jul 16 22:14:26.280903 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:14:26.280832    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1\": container with ID starting with 3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1 not found: ID does not exist" containerID="3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"
Jul 16 22:14:26.280903 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.280862    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1"} err="failed to get container status \"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1\": rpc error: code = NotFound desc = could not find container \"3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1\": container with ID starting with 3adcf7b5030fd4f54b0e95fc57079923d525d0b765395ab82bce53520eed3bb1 not found: ID does not exist"
Jul 16 22:14:26.287419 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.287393    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:14:26.292519 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.292497    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejbt5j"]
Jul 16 22:14:26.319942 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:26.319916    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" path="/var/lib/kubelet/pods/c52f4ff3-456e-4c21-8a07-f1d295228b9a/volumes"
Jul 16 22:14:34.101577 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101542    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"]
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101914    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="storage-initializer"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101926    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="storage-initializer"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101939    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="storage-initializer"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101944    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="storage-initializer"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101950    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101955    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101967    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="main"
Jul 16 22:14:34.101985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.101973    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="main"
Jul 16 22:14:34.102233 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.102044    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0e55ef-d33b-4cf6-abf8-887db4179b0e" containerName="main"
Jul 16 22:14:34.102233 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.102053    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52f4ff3-456e-4c21-8a07-f1d295228b9a" containerName="main"
Jul 16 22:14:34.105070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.105043    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:34.107671 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.107649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:14:34.107934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.107913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"]
Jul 16 22:14:34.108750 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.108727    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:14:34.108896 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.108727    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:14:34.215508 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.215467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl99\" (UniqueName: \"kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:34.316151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.316112    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl99\" (UniqueName: \"kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:34.325633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:34.325598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl99\" (UniqueName: \"kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:36.969720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.969682    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:14:36.973669 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.973651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:36.977359 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.977331    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 16 22:14:36.977490 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.977393    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-zth96\""
Jul 16 22:14:36.977490 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.977459    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:14:36.983592 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:36.983568    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:14:37.034387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.034347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.034583 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.034451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.034583 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.034485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.034583 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.034541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhv5\" (UniqueName: \"kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135086 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135264 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135264 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135264 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhv5\" (UniqueName: \"kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135435 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.135509 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.135488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.137725 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.137709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.147631 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.147600    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhv5\" (UniqueName: \"kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5\") pod \"custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.286097 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.285991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:37.430620 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:37.430589    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:14:37.433142 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:14:37.433104    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d36388_7dbc_4f94_8af7_f01885e06dc8.slice/crio-cd551db653f4cd3afd40791ba5c25727d642e3f8091d6782f81fd570f52d8ec7 WatchSource:0}: Error finding container cd551db653f4cd3afd40791ba5c25727d642e3f8091d6782f81fd570f52d8ec7: Status 404 returned error can't find the container with id cd551db653f4cd3afd40791ba5c25727d642e3f8091d6782f81fd570f52d8ec7
Jul 16 22:14:38.320587 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:38.320551    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerID="aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee" exitCode=0
Jul 16 22:14:38.321140 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:38.321114    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" event={"ID":"c2d36388-7dbc-4f94-8af7-f01885e06dc8","Type":"ContainerDied","Data":"aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee"}
Jul 16 22:14:38.321185 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:38.321170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" event={"ID":"c2d36388-7dbc-4f94-8af7-f01885e06dc8","Type":"ContainerStarted","Data":"cd551db653f4cd3afd40791ba5c25727d642e3f8091d6782f81fd570f52d8ec7"}
Jul 16 22:14:39.327660 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:39.327620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" event={"ID":"c2d36388-7dbc-4f94-8af7-f01885e06dc8","Type":"ContainerStarted","Data":"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"}
Jul 16 22:14:39.328246 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:39.327811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:14:39.348204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:39.348147    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" podStartSLOduration=2.6132281219999998 podStartE2EDuration="3.348130967s" totalImagesPullingTime="0s" totalInitContainerRuntime="734.902845ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:39.345889112 +0000 UTC m=+1479.441400430" watchObservedRunningTime="2026-07-16 22:14:39.348130967 +0000 UTC m=+1479.443642283"
Jul 16 22:14:40.761346 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:40.761308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:40.768198 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:40.768163    2566 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:40.862560 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:40.862521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:40.862743 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:40.862633    2566 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:40.865650 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:40.865626    2566 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") DevicePath \"csi-fc6866da7516f983eb6ae54f187c5f56b60af9f037d4be79aa79fd45de44ab89\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:41.090838 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:41.090779    2566 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a0f5e3d1545eedd31c6a3b629df7df5a5400a46392625f738ffdd5e141917306/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:41.107930 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:41.107902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"e2e-pvc-model-download-w44vk\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:41.315207 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:41.315172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:41.453654 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:41.453620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"]
Jul 16 22:14:41.456194 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:14:41.456161    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8ff498_5f38_437c_aa39_02e8bc124806.slice/crio-6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804 WatchSource:0}: Error finding container 6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804: Status 404 returned error can't find the container with id 6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804
Jul 16 22:14:42.354787 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:42.354744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk" event={"ID":"5f8ff498-5f38-437c-aa39-02e8bc124806","Type":"ContainerStarted","Data":"d6b973c6fcc3dba339b2042270965640256f67ff7a6c955f7dbdbd0dd18ad7e8"}
Jul 16 22:14:42.354787 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:42.354835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk" event={"ID":"5f8ff498-5f38-437c-aa39-02e8bc124806","Type":"ContainerStarted","Data":"6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804"}
Jul 16 22:14:42.599570 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:42.599445    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk" podStartSLOduration=11.599417879 podStartE2EDuration="11.599417879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:42.389472715 +0000 UTC m=+1482.484984031" watchObservedRunningTime="2026-07-16 22:14:42.599417879 +0000 UTC m=+1482.694929196"
Jul 16 22:14:48.383305 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:48.383269    2566 generic.go:356] "Generic (PLEG): container finished" podID="5f8ff498-5f38-437c-aa39-02e8bc124806" containerID="d6b973c6fcc3dba339b2042270965640256f67ff7a6c955f7dbdbd0dd18ad7e8" exitCode=0
Jul 16 22:14:48.383688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:48.383355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk" event={"ID":"5f8ff498-5f38-437c-aa39-02e8bc124806","Type":"ContainerDied","Data":"d6b973c6fcc3dba339b2042270965640256f67ff7a6c955f7dbdbd0dd18ad7e8"}
Jul 16 22:14:49.568578 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.568550    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:14:49.639437 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.639339    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgl99\" (UniqueName: \"kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99\") pod \"5f8ff498-5f38-437c-aa39-02e8bc124806\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") "
Jul 16 22:14:49.641535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.641504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99" (OuterVolumeSpecName: "kube-api-access-jgl99") pod "5f8ff498-5f38-437c-aa39-02e8bc124806" (UID: "5f8ff498-5f38-437c-aa39-02e8bc124806"). InnerVolumeSpecName "kube-api-access-jgl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:14:49.740258 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.740225    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") pod \"5f8ff498-5f38-437c-aa39-02e8bc124806\" (UID: \"5f8ff498-5f38-437c-aa39-02e8bc124806\") "
Jul 16 22:14:49.740422 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.740378    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgl99\" (UniqueName: \"kubernetes.io/projected/5f8ff498-5f38-437c-aa39-02e8bc124806-kube-api-access-jgl99\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:14:49.744288 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.744262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8" (OuterVolumeSpecName: "model-storage") pod "5f8ff498-5f38-437c-aa39-02e8bc124806" (UID: "5f8ff498-5f38-437c-aa39-02e8bc124806"). InnerVolumeSpecName "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 22:14:49.841200 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.841167    2566 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-143-100.ec2.internal\" "
Jul 16 22:14:49.951951 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:49.951927    2566 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8") on node "ip-10-0-143-100.ec2.internal"
Jul 16 22:14:50.041978 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:50.041939    2566 reconciler_common.go:299] "Volume detached for volume \"pvc-04ad49c2-6cf9-47f2-b0f1-d4607b16c200\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0c32052d07cfa18c8\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"csi-fc6866da7516f983eb6ae54f187c5f56b60af9f037d4be79aa79fd45de44ab89\""
Jul 16 22:14:50.393043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:50.393011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk" event={"ID":"5f8ff498-5f38-437c-aa39-02e8bc124806","Type":"ContainerDied","Data":"6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804"}
Jul 16 22:14:50.393043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:50.393050    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fe316d04315528ee41027d9932683667b49c7232448eb1b5531eaa293cd5804"
Jul 16 22:14:50.393222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:14:50.393025    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-w44vk"
Jul 16 22:15:00.355643 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.355615    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:15:00.356463 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.356447    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:15:00.936002 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.935945    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:15:00.936331 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.936312    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8ff498-5f38-437c-aa39-02e8bc124806" containerName="storage-initializer"
Jul 16 22:15:00.936331 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.936333    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8ff498-5f38-437c-aa39-02e8bc124806" containerName="storage-initializer"
Jul 16 22:15:00.936516 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.936503    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8ff498-5f38-437c-aa39-02e8bc124806" containerName="storage-initializer"
Jul 16 22:15:00.941027 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.941003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:00.944810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.944778    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-7q4tr\""
Jul 16 22:15:00.944970 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.944889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 16 22:15:00.950718 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:00.950688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:15:01.035443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.035405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.035443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.035445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.035652 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.035479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q8nz\" (UniqueName: \"kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.136086 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.136051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.136261 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.136106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q8nz\" (UniqueName: \"kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.136261 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.136171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.136438 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.136411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.138695 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.138675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.145190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.145159    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q8nz\" (UniqueName: \"kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.253558 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.253524    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:01.394929 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.394893    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:15:01.398165 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:15:01.398137    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51995326_d23b_4afc_b71b_d26e6cb45d27.slice/crio-945e2321bfd0c4e45911f20439e6e7cce9749c0baaca53703148ad91b37298b6 WatchSource:0}: Error finding container 945e2321bfd0c4e45911f20439e6e7cce9749c0baaca53703148ad91b37298b6: Status 404 returned error can't find the container with id 945e2321bfd0c4e45911f20439e6e7cce9749c0baaca53703148ad91b37298b6
Jul 16 22:15:01.447089 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:01.447062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" event={"ID":"51995326-d23b-4afc-b71b-d26e6cb45d27","Type":"ContainerStarted","Data":"945e2321bfd0c4e45911f20439e6e7cce9749c0baaca53703148ad91b37298b6"}
Jul 16 22:15:02.462006 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:02.461963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" event={"ID":"51995326-d23b-4afc-b71b-d26e6cb45d27","Type":"ContainerStarted","Data":"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"}
Jul 16 22:15:02.462434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:02.462151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:15:02.480562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:02.480510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" podStartSLOduration=2.480495578 podStartE2EDuration="2.480495578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:15:02.480041449 +0000 UTC m=+1502.575552766" watchObservedRunningTime="2026-07-16 22:15:02.480495578 +0000 UTC m=+1502.576006936"
Jul 16 22:15:10.334405 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:10.334372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:15:33.468650 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:15:33.468573    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:16:54.961377 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:16:54.961299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:17:00.334635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:00.334599    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:17:09.986321 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:09.986255    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="main" containerID="cri-o://d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052" gracePeriod=45
Jul 16 22:17:10.240706 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.240638    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:17:10.343763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.343727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs\") pod \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") "
Jul 16 22:17:10.343952 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.343773    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location\") pod \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") "
Jul 16 22:17:10.343952 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.343839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvhv5\" (UniqueName: \"kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5\") pod \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") "
Jul 16 22:17:10.343952 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.343871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds\") pod \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\" (UID: \"c2d36388-7dbc-4f94-8af7-f01885e06dc8\") "
Jul 16 22:17:10.344245 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.344218    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c2d36388-7dbc-4f94-8af7-f01885e06dc8" (UID: "c2d36388-7dbc-4f94-8af7-f01885e06dc8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:10.344621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.344592    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2d36388-7dbc-4f94-8af7-f01885e06dc8" (UID: "c2d36388-7dbc-4f94-8af7-f01885e06dc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:10.346156 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.346125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2d36388-7dbc-4f94-8af7-f01885e06dc8" (UID: "c2d36388-7dbc-4f94-8af7-f01885e06dc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:17:10.346156 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.346133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5" (OuterVolumeSpecName: "kube-api-access-kvhv5") pod "c2d36388-7dbc-4f94-8af7-f01885e06dc8" (UID: "c2d36388-7dbc-4f94-8af7-f01885e06dc8"). InnerVolumeSpecName "kube-api-access-kvhv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:10.444785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.444748    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:10.444785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.444781    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:10.444785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.444794    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhv5\" (UniqueName: \"kubernetes.io/projected/c2d36388-7dbc-4f94-8af7-f01885e06dc8-kube-api-access-kvhv5\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:10.445091 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:10.444805    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c2d36388-7dbc-4f94-8af7-f01885e06dc8-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:11.068599 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.068559    2566 generic.go:356] "Generic (PLEG): container finished" podID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerID="d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052" exitCode=0
Jul 16 22:17:11.069092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.068608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"
Jul 16 22:17:11.069092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.068618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" event={"ID":"c2d36388-7dbc-4f94-8af7-f01885e06dc8","Type":"ContainerDied","Data":"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"}
Jul 16 22:17:11.069092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.068659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts" event={"ID":"c2d36388-7dbc-4f94-8af7-f01885e06dc8","Type":"ContainerDied","Data":"cd551db653f4cd3afd40791ba5c25727d642e3f8091d6782f81fd570f52d8ec7"}
Jul 16 22:17:11.069092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.068679    2566 scope.go:117] "RemoveContainer" containerID="d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"
Jul 16 22:17:11.078249 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.078230    2566 scope.go:117] "RemoveContainer" containerID="aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee"
Jul 16 22:17:11.086862 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.086841    2566 scope.go:117] "RemoveContainer" containerID="d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"
Jul 16 22:17:11.087153 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:17:11.087133    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052\": container with ID starting with d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052 not found: ID does not exist" containerID="d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"
Jul 16 22:17:11.087213 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.087162    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052"} err="failed to get container status \"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052\": rpc error: code = NotFound desc = could not find container \"d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052\": container with ID starting with d30afb828aa12332ac54a09b7832c20173854a6b6c2f0f793607754909160052 not found: ID does not exist"
Jul 16 22:17:11.087213 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.087182    2566 scope.go:117] "RemoveContainer" containerID="aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee"
Jul 16 22:17:11.087467 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:17:11.087447    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee\": container with ID starting with aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee not found: ID does not exist" containerID="aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee"
Jul 16 22:17:11.087566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.087473    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee"} err="failed to get container status \"aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee\": rpc error: code = NotFound desc = could not find container \"aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee\": container with ID starting with aa735949ccf7ccf337c34390d3a6950be6f3a20c60487c652f495340480289ee not found: ID does not exist"
Jul 16 22:17:11.092978 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.092952    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:17:11.097472 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:11.097437    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-6c56cf97vk2ts"]
Jul 16 22:17:12.319740 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:12.319708    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" path="/var/lib/kubelet/pods/c2d36388-7dbc-4f94-8af7-f01885e06dc8/volumes"
Jul 16 22:17:41.261345 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261306    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:17:41.261854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261643    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="storage-initializer"
Jul 16 22:17:41.261854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261655    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="storage-initializer"
Jul 16 22:17:41.261854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261675    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="main"
Jul 16 22:17:41.261854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261680    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="main"
Jul 16 22:17:41.261854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.261738    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d36388-7dbc-4f94-8af7-f01885e06dc8" containerName="main"
Jul 16 22:17:41.264707 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.264686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.267298 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.267276    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:17:41.268448 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.268425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-7qcdh\""
Jul 16 22:17:41.268578 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.268467    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 16 22:17:41.275431 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.275407    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:17:41.327695 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.327646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.327938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.327737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.327938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.327768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.327938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.327793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpsp2\" (UniqueName: \"kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.428963 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.428919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.428963 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.428960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.429189 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.428985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpsp2\" (UniqueName: \"kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.429189 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.429038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.429456 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.429430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.429496 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.429451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.431986 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.431953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.437676 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.437647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpsp2\" (UniqueName: \"kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2\") pod \"router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.574976 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.574889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:41.718245 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:41.718185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:17:41.720433 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:17:41.720407    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f372954_b286_4d54_899c_77836d3afcc3.slice/crio-54e78056176cb6508c68bc70c3ba82c5a1f3d83538b20b424125b303686717a3 WatchSource:0}: Error finding container 54e78056176cb6508c68bc70c3ba82c5a1f3d83538b20b424125b303686717a3: Status 404 returned error can't find the container with id 54e78056176cb6508c68bc70c3ba82c5a1f3d83538b20b424125b303686717a3
Jul 16 22:17:42.220906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:42.220861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerStarted","Data":"64b27e10c1651ab1272bbf9fb432b69f580b6ee44f97ae0a00f4d2f04b652128"}
Jul 16 22:17:42.221100 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:42.220920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerStarted","Data":"54e78056176cb6508c68bc70c3ba82c5a1f3d83538b20b424125b303686717a3"}
Jul 16 22:17:43.227588 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:43.227544    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f372954-b286-4d54-899c-77836d3afcc3" containerID="64b27e10c1651ab1272bbf9fb432b69f580b6ee44f97ae0a00f4d2f04b652128" exitCode=0
Jul 16 22:17:43.228038 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:43.227678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerDied","Data":"64b27e10c1651ab1272bbf9fb432b69f580b6ee44f97ae0a00f4d2f04b652128"}
Jul 16 22:17:43.332152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:43.332115    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:17:43.469185 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:43.469149    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:17:44.235562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:44.235528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerStarted","Data":"6077d2981b369adabd2173b0d6fb8f5e197458c685f10ab67f77c91fb0d29651"}
Jul 16 22:17:44.235986 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:44.235617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:17:44.254589 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:44.254530    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" podStartSLOduration=2.478488831 podStartE2EDuration="3.254508556s" totalImagesPullingTime="0s" totalInitContainerRuntime="776.019725ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:44.252609629 +0000 UTC m=+1664.348120947" watchObservedRunningTime="2026-07-16 22:17:44.254508556 +0000 UTC m=+1664.350019874"
Jul 16 22:17:53.469091 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:53.469050    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:17:58.355335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.355286    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main" containerID="cri-o://f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1" gracePeriod=45
Jul 16 22:17:58.613276 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.613212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:17:58.778476 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.778437    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs\") pod \"51995326-d23b-4afc-b71b-d26e6cb45d27\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") "
Jul 16 22:17:58.778476 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.778482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q8nz\" (UniqueName: \"kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz\") pod \"51995326-d23b-4afc-b71b-d26e6cb45d27\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") "
Jul 16 22:17:58.778687 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.778566    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds\") pod \"51995326-d23b-4afc-b71b-d26e6cb45d27\" (UID: \"51995326-d23b-4afc-b71b-d26e6cb45d27\") "
Jul 16 22:17:58.778904 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.778875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "51995326-d23b-4afc-b71b-d26e6cb45d27" (UID: "51995326-d23b-4afc-b71b-d26e6cb45d27"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:17:58.780878 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.780851    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz" (OuterVolumeSpecName: "kube-api-access-8q8nz") pod "51995326-d23b-4afc-b71b-d26e6cb45d27" (UID: "51995326-d23b-4afc-b71b-d26e6cb45d27"). InnerVolumeSpecName "kube-api-access-8q8nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:17:58.780878 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.780857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51995326-d23b-4afc-b71b-d26e6cb45d27" (UID: "51995326-d23b-4afc-b71b-d26e6cb45d27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:17:58.880126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.880033    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51995326-d23b-4afc-b71b-d26e6cb45d27-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:58.880126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.880066    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51995326-d23b-4afc-b71b-d26e6cb45d27-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:58.880126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:58.880079    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q8nz\" (UniqueName: \"kubernetes.io/projected/51995326-d23b-4afc-b71b-d26e6cb45d27-kube-api-access-8q8nz\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:17:59.306806 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.306771    2566 generic.go:356] "Generic (PLEG): container finished" podID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerID="f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1" exitCode=0
Jul 16 22:17:59.306993 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.306812    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"
Jul 16 22:17:59.306993 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.306851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" event={"ID":"51995326-d23b-4afc-b71b-d26e6cb45d27","Type":"ContainerDied","Data":"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"}
Jul 16 22:17:59.306993 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.306902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9" event={"ID":"51995326-d23b-4afc-b71b-d26e6cb45d27","Type":"ContainerDied","Data":"945e2321bfd0c4e45911f20439e6e7cce9749c0baaca53703148ad91b37298b6"}
Jul 16 22:17:59.306993 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.306922    2566 scope.go:117] "RemoveContainer" containerID="f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"
Jul 16 22:17:59.316984 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.316967    2566 scope.go:117] "RemoveContainer" containerID="f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"
Jul 16 22:17:59.317268 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:17:59.317248    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1\": container with ID starting with f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1 not found: ID does not exist" containerID="f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"
Jul 16 22:17:59.317343 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.317276    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1"} err="failed to get container status \"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1\": rpc error: code = NotFound desc = could not find container \"f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1\": container with ID starting with f3723692eb35f2f7fcb1538fe9143a3944b2d128ece1871b88a2acf6ca7145e1 not found: ID does not exist"
Jul 16 22:17:59.327909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.327860    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:17:59.329690 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:17:59.329667    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-scherndv9"]
Jul 16 22:18:00.319660 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:00.319630    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" path="/var/lib/kubelet/pods/51995326-d23b-4afc-b71b-d26e6cb45d27/volumes"
Jul 16 22:18:09.410627 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.410589    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:18:09.411116 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.410957    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main"
Jul 16 22:18:09.411116 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.410970    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main"
Jul 16 22:18:09.411116 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.411043    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51995326-d23b-4afc-b71b-d26e6cb45d27" containerName="main"
Jul 16 22:18:09.415560 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.415540    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.419426 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.419407    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 16 22:18:09.422270 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.422066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-2gjc8\""
Jul 16 22:18:09.432385 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.432362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:18:09.474942 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.474903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpbv\" (UniqueName: \"kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.475108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.475015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.475108 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.475091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.576238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.576194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.576431 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.576253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.576431 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.576286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpbv\" (UniqueName: \"kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.576728 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.576707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.578924 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.578905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.591323 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.591298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpbv\" (UniqueName: \"kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.724806 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.724773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:09.890202 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:09.890065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:18:09.893072 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:18:09.893038    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b2aa6c_4409_48ac_92e4_5a4076f79a58.slice/crio-5da023582fe630c8b6a8ecd4447242c114a445ecd52a52520b990add9bcd4be7 WatchSource:0}: Error finding container 5da023582fe630c8b6a8ecd4447242c114a445ecd52a52520b990add9bcd4be7: Status 404 returned error can't find the container with id 5da023582fe630c8b6a8ecd4447242c114a445ecd52a52520b990add9bcd4be7
Jul 16 22:18:10.362800 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:10.362764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" event={"ID":"68b2aa6c-4409-48ac-92e4-5a4076f79a58","Type":"ContainerStarted","Data":"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"}
Jul 16 22:18:10.363017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:10.362837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" event={"ID":"68b2aa6c-4409-48ac-92e4-5a4076f79a58","Type":"ContainerStarted","Data":"5da023582fe630c8b6a8ecd4447242c114a445ecd52a52520b990add9bcd4be7"}
Jul 16 22:18:10.363088 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:10.363019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:18:10.382786 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:10.382731    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" podStartSLOduration=1.382717461 podStartE2EDuration="1.382717461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:10.381028728 +0000 UTC m=+1690.476540048" watchObservedRunningTime="2026-07-16 22:18:10.382717461 +0000 UTC m=+1690.478228778"
Jul 16 22:18:15.242276 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:15.242250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:18:41.369481 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:18:41.369452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:19:51.685380 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:19:51.685303    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:19:55.243261 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:19:55.243226    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:20:00.384982 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:00.384955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:20:00.387619 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:00.387593    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:20:05.242361 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:05.242320    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:20:06.705654 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.705606    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main" containerID="cri-o://6077d2981b369adabd2173b0d6fb8f5e197458c685f10ab67f77c91fb0d29651" gracePeriod=45
Jul 16 22:20:06.904945 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.904907    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f372954-b286-4d54-899c-77836d3afcc3" containerID="6077d2981b369adabd2173b0d6fb8f5e197458c685f10ab67f77c91fb0d29651" exitCode=0
Jul 16 22:20:06.905115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.904980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerDied","Data":"6077d2981b369adabd2173b0d6fb8f5e197458c685f10ab67f77c91fb0d29651"}
Jul 16 22:20:06.958783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.958701    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:20:06.986738 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.986708    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs\") pod \"1f372954-b286-4d54-899c-77836d3afcc3\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") "
Jul 16 22:20:06.986964 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.986755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpsp2\" (UniqueName: \"kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2\") pod \"1f372954-b286-4d54-899c-77836d3afcc3\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") "
Jul 16 22:20:06.986964 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.986806    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location\") pod \"1f372954-b286-4d54-899c-77836d3afcc3\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") "
Jul 16 22:20:06.986964 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.986857    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds\") pod \"1f372954-b286-4d54-899c-77836d3afcc3\" (UID: \"1f372954-b286-4d54-899c-77836d3afcc3\") "
Jul 16 22:20:06.987148 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.987122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1f372954-b286-4d54-899c-77836d3afcc3" (UID: "1f372954-b286-4d54-899c-77836d3afcc3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:06.987512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.987483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f372954-b286-4d54-899c-77836d3afcc3" (UID: "1f372954-b286-4d54-899c-77836d3afcc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:06.989107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.989077    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2" (OuterVolumeSpecName: "kube-api-access-cpsp2") pod "1f372954-b286-4d54-899c-77836d3afcc3" (UID: "1f372954-b286-4d54-899c-77836d3afcc3"). InnerVolumeSpecName "kube-api-access-cpsp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:20:06.989204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:06.989152    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f372954-b286-4d54-899c-77836d3afcc3" (UID: "1f372954-b286-4d54-899c-77836d3afcc3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:20:07.088222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.088183    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f372954-b286-4d54-899c-77836d3afcc3-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:20:07.088222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.088222    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpsp2\" (UniqueName: \"kubernetes.io/projected/1f372954-b286-4d54-899c-77836d3afcc3-kube-api-access-cpsp2\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:20:07.088222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.088235    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:20:07.088480 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.088249    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1f372954-b286-4d54-899c-77836d3afcc3-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:20:07.912123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.912090    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"
Jul 16 22:20:07.912580 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.912096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k" event={"ID":"1f372954-b286-4d54-899c-77836d3afcc3","Type":"ContainerDied","Data":"54e78056176cb6508c68bc70c3ba82c5a1f3d83538b20b424125b303686717a3"}
Jul 16 22:20:07.912580 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.912224    2566 scope.go:117] "RemoveContainer" containerID="6077d2981b369adabd2173b0d6fb8f5e197458c685f10ab67f77c91fb0d29651"
Jul 16 22:20:07.921544 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.921524    2566 scope.go:117] "RemoveContainer" containerID="64b27e10c1651ab1272bbf9fb432b69f580b6ee44f97ae0a00f4d2f04b652128"
Jul 16 22:20:07.935126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.935101    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:20:07.940432 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:07.940408    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-55c98c487b-tzm6k"]
Jul 16 22:20:08.319242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:08.319207    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f372954-b286-4d54-899c-77836d3afcc3" path="/var/lib/kubelet/pods/1f372954-b286-4d54-899c-77836d3afcc3/volumes"
Jul 16 22:20:20.652549 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652513    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:20:20.653225 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652865    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="storage-initializer"
Jul 16 22:20:20.653225 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652876    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="storage-initializer"
Jul 16 22:20:20.653225 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652897    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main"
Jul 16 22:20:20.653225 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main"
Jul 16 22:20:20.653225 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.652983    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f372954-b286-4d54-899c-77836d3afcc3" containerName="main"
Jul 16 22:20:20.657431 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.657412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.660579 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.660560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 16 22:20:20.661035 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.661021    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:20:20.662017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.662002    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-hl94h\""
Jul 16 22:20:20.676924 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.676900    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:20:20.703985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.703960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.704133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.704005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv5dq\" (UniqueName: \"kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.704133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.704065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.704133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.704085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.804906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.804867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.805065 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.804912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.805065 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.804959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.805065 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.804993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv5dq\" (UniqueName: \"kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.805258 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.805234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.805349 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.805319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.807884 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.807865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.815752 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.815732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv5dq\" (UniqueName: \"kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:20.966683 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:20.966643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:21.111476 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:21.111408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:20:21.113870 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:20:21.113842    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b3bfde_f2d4_49d6_abcc_35cf69e1e32e.slice/crio-5cce61be02102a0d381cc75425a483a1137078b28925b22cf1ee7c6acadfc3ae WatchSource:0}: Error finding container 5cce61be02102a0d381cc75425a483a1137078b28925b22cf1ee7c6acadfc3ae: Status 404 returned error can't find the container with id 5cce61be02102a0d381cc75425a483a1137078b28925b22cf1ee7c6acadfc3ae
Jul 16 22:20:21.981991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:21.981955    2566 generic.go:356] "Generic (PLEG): container finished" podID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerID="71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0" exitCode=0
Jul 16 22:20:21.982374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:21.982044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" event={"ID":"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e","Type":"ContainerDied","Data":"71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0"}
Jul 16 22:20:21.982374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:21.982104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" event={"ID":"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e","Type":"ContainerStarted","Data":"5cce61be02102a0d381cc75425a483a1137078b28925b22cf1ee7c6acadfc3ae"}
Jul 16 22:20:22.989459 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:22.989417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" event={"ID":"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e","Type":"ContainerStarted","Data":"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"}
Jul 16 22:20:22.989848 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:22.989520    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:20:23.016380 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:23.016318    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" podStartSLOduration=2.297326872 podStartE2EDuration="3.016300778s" totalImagesPullingTime="0s" totalInitContainerRuntime="718.973906ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:23.015528475 +0000 UTC m=+1823.111039847" watchObservedRunningTime="2026-07-16 22:20:23.016300778 +0000 UTC m=+1823.111812097"
Jul 16 22:20:53.995957 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:20:53.995928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:21:16.729686 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:16.729650    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:21:21.369937 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:21.369847    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:21:31.369607 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:31.369563    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:21:31.750418 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:31.750375    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main" containerID="cri-o://bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285" gracePeriod=45
Jul 16 22:21:32.005974 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.005903    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:21:32.106745 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.106709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhpbv\" (UniqueName: \"kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv\") pod \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") "
Jul 16 22:21:32.106973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.106792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs\") pod \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") "
Jul 16 22:21:32.106973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.106906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds\") pod \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\" (UID: \"68b2aa6c-4409-48ac-92e4-5a4076f79a58\") "
Jul 16 22:21:32.107138 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.107116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "68b2aa6c-4409-48ac-92e4-5a4076f79a58" (UID: "68b2aa6c-4409-48ac-92e4-5a4076f79a58"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:32.108999 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.108974    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv" (OuterVolumeSpecName: "kube-api-access-nhpbv") pod "68b2aa6c-4409-48ac-92e4-5a4076f79a58" (UID: "68b2aa6c-4409-48ac-92e4-5a4076f79a58"). InnerVolumeSpecName "kube-api-access-nhpbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:32.108999 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.108976    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68b2aa6c-4409-48ac-92e4-5a4076f79a58" (UID: "68b2aa6c-4409-48ac-92e4-5a4076f79a58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:21:32.208322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.208282    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhpbv\" (UniqueName: \"kubernetes.io/projected/68b2aa6c-4409-48ac-92e4-5a4076f79a58-kube-api-access-nhpbv\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.208322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.208313    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.208322 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.208322    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/68b2aa6c-4409-48ac-92e4-5a4076f79a58-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:21:32.318096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.318000    2566 generic.go:356] "Generic (PLEG): container finished" podID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerID="bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285" exitCode=0
Jul 16 22:21:32.318273 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.318113    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"
Jul 16 22:21:32.320373 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.320342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" event={"ID":"68b2aa6c-4409-48ac-92e4-5a4076f79a58","Type":"ContainerDied","Data":"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"}
Jul 16 22:21:32.320491 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.320393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn" event={"ID":"68b2aa6c-4409-48ac-92e4-5a4076f79a58","Type":"ContainerDied","Data":"5da023582fe630c8b6a8ecd4447242c114a445ecd52a52520b990add9bcd4be7"}
Jul 16 22:21:32.320491 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.320421    2566 scope.go:117] "RemoveContainer" containerID="bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"
Jul 16 22:21:32.330376 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.330344    2566 scope.go:117] "RemoveContainer" containerID="bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"
Jul 16 22:21:32.330670 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:21:32.330647    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285\": container with ID starting with bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285 not found: ID does not exist" containerID="bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"
Jul 16 22:21:32.330746 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.330678    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285"} err="failed to get container status \"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285\": rpc error: code = NotFound desc = could not find container \"bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285\": container with ID starting with bc1f59701fd0dc9e998bb973154140915d83df69bdcd2aa4fd4480f71215b285 not found: ID does not exist"
Jul 16 22:21:32.344423 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.344395    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:21:32.348806 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:32.348781    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schehvsgn"]
Jul 16 22:21:34.319720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:34.319680    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" path="/var/lib/kubelet/pods/68b2aa6c-4409-48ac-92e4-5a4076f79a58/volumes"
Jul 16 22:21:49.285668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.285636    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:21:49.288075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.286053    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main"
Jul 16 22:21:49.288075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.286067    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main"
Jul 16 22:21:49.288075 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.286150    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b2aa6c-4409-48ac-92e4-5a4076f79a58" containerName="main"
Jul 16 22:21:49.289009 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.288993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.295466 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.295442    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-ws2z7\""
Jul 16 22:21:49.295618 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.295539    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 16 22:21:49.305728 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.305701    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:21:49.364244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.364205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.364244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.364249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.364467 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.364270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5wl8\" (UniqueName: \"kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.465367 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.465333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.465578 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.465372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wl8\" (UniqueName: \"kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.465578 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.465455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.465791 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.465773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.468186 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.468164    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.478181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.478155    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wl8\" (UniqueName: \"kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.598873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.598763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:49.741080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:49.741056    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:21:49.743321 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:21:49.743295    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d18b99_f4bc_45f2_99e1_164e303fcbc4.slice/crio-5c60ae7bbed373014b81332d8bd27003ac13a38e7a1a3fbd67e03a323c68512a WatchSource:0}: Error finding container 5c60ae7bbed373014b81332d8bd27003ac13a38e7a1a3fbd67e03a323c68512a: Status 404 returned error can't find the container with id 5c60ae7bbed373014b81332d8bd27003ac13a38e7a1a3fbd67e03a323c68512a
Jul 16 22:21:50.401377 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:50.401346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" event={"ID":"b2d18b99-f4bc-45f2-99e1-164e303fcbc4","Type":"ContainerStarted","Data":"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"}
Jul 16 22:21:50.401777 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:50.401392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" event={"ID":"b2d18b99-f4bc-45f2-99e1-164e303fcbc4","Type":"ContainerStarted","Data":"5c60ae7bbed373014b81332d8bd27003ac13a38e7a1a3fbd67e03a323c68512a"}
Jul 16 22:21:50.401777 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:50.401440    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:21:50.442402 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:21:50.442348    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" podStartSLOduration=1.4423311079999999 podStartE2EDuration="1.442331108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:50.440517366 +0000 UTC m=+1910.536028683" watchObservedRunningTime="2026-07-16 22:21:50.442331108 +0000 UTC m=+1910.537842452"
Jul 16 22:22:21.407096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:22:21.407067    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:23:53.996397 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:23:53.996357    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:23:54.064606 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:23:54.064572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:24:03.996492 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:03.996451    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:24:09.086080 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.086016    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main" containerID="cri-o://8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11" gracePeriod=45
Jul 16 22:24:09.344392 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.344327    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:24:09.369855 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.369787    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs\") pod \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") "
Jul 16 22:24:09.370055 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.369863    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location\") pod \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") "
Jul 16 22:24:09.370055 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.369967    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv5dq\" (UniqueName: \"kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq\") pod \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") "
Jul 16 22:24:09.370198 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.370057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds\") pod \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\" (UID: \"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e\") "
Jul 16 22:24:09.370389 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.370346    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" (UID: "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:09.370762 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.370736    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" (UID: "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:24:09.372427 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.372402    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" (UID: "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:24:09.372541 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.372431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq" (OuterVolumeSpecName: "kube-api-access-zv5dq") pod "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" (UID: "21b3bfde-f2d4-49d6-abcc-35cf69e1e32e"). InnerVolumeSpecName "kube-api-access-zv5dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:24:09.471151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.471112    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:24:09.471151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.471146    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:24:09.471151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.471157    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv5dq\" (UniqueName: \"kubernetes.io/projected/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-kube-api-access-zv5dq\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:24:09.471151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:09.471165    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:24:10.069236 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.069194    2566 generic.go:356] "Generic (PLEG): container finished" podID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerID="8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11" exitCode=0
Jul 16 22:24:10.069462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.069261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"
Jul 16 22:24:10.069462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.069281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" event={"ID":"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e","Type":"ContainerDied","Data":"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"}
Jul 16 22:24:10.069462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.069337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj" event={"ID":"21b3bfde-f2d4-49d6-abcc-35cf69e1e32e","Type":"ContainerDied","Data":"5cce61be02102a0d381cc75425a483a1137078b28925b22cf1ee7c6acadfc3ae"}
Jul 16 22:24:10.069462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.069362    2566 scope.go:117] "RemoveContainer" containerID="8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"
Jul 16 22:24:10.078747 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.078722    2566 scope.go:117] "RemoveContainer" containerID="71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0"
Jul 16 22:24:10.088726 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.088498    2566 scope.go:117] "RemoveContainer" containerID="8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"
Jul 16 22:24:10.089057 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:24:10.088766    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11\": container with ID starting with 8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11 not found: ID does not exist" containerID="8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"
Jul 16 22:24:10.089057 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.088795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11"} err="failed to get container status \"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11\": rpc error: code = NotFound desc = could not find container \"8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11\": container with ID starting with 8e3f0e0294eeba66c95064a4f9b25d72bcb0ac20db8ba5f8fd77e3177ad1eb11 not found: ID does not exist"
Jul 16 22:24:10.089057 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.088837    2566 scope.go:117] "RemoveContainer" containerID="71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0"
Jul 16 22:24:10.089224 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:24:10.089079    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0\": container with ID starting with 71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0 not found: ID does not exist" containerID="71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0"
Jul 16 22:24:10.089224 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.089101    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0"} err="failed to get container status \"71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0\": rpc error: code = NotFound desc = could not find container \"71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0\": container with ID starting with 71f745aec1f22540ea3395cf63c2007ccef4442d7281977cfb9817fff16e15e0 not found: ID does not exist"
Jul 16 22:24:10.096395 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.096369    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:24:10.104446 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.104420    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schewjstj"]
Jul 16 22:24:10.319651 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:10.319569    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" path="/var/lib/kubelet/pods/21b3bfde-f2d4-49d6-abcc-35cf69e1e32e/volumes"
Jul 16 22:24:23.386195 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386103    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:24:23.386674 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386655    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="storage-initializer"
Jul 16 22:24:23.386720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386676    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="storage-initializer"
Jul 16 22:24:23.386720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386694    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main"
Jul 16 22:24:23.386720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386703    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main"
Jul 16 22:24:23.386869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.386851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b3bfde-f2d4-49d6-abcc-35cf69e1e32e" containerName="main"
Jul 16 22:24:23.392122 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.392098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.395409 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.395385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:24:23.395535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.395383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-28jsw\""
Jul 16 22:24:23.395535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.395464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 16 22:24:23.403545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.403522    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:24:23.500024 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.499983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.500024 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.500029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmx4c\" (UniqueName: \"kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.500300 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.500051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.500300 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.500214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.601514 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.601705 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.601705 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmx4c\" (UniqueName: \"kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.601705 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.601956 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.602025 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.601951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.604318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.604287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.624962 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.624935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmx4c\" (UniqueName: \"kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.702699 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.702665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:23.844196 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:23.844171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:24:23.846224 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:24:23.846196    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1c5459_6acc_4c45_b257_cb9068fffa20.slice/crio-4712e5e53eaecc19a2ae4fa4020b514078b84764970bc5032c7420fb6baef60f WatchSource:0}: Error finding container 4712e5e53eaecc19a2ae4fa4020b514078b84764970bc5032c7420fb6baef60f: Status 404 returned error can't find the container with id 4712e5e53eaecc19a2ae4fa4020b514078b84764970bc5032c7420fb6baef60f
Jul 16 22:24:24.135571 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:24.135485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerStarted","Data":"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"}
Jul 16 22:24:24.135571 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:24.135543    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerStarted","Data":"4712e5e53eaecc19a2ae4fa4020b514078b84764970bc5032c7420fb6baef60f"}
Jul 16 22:24:25.144992 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:25.144951    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerID="838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816" exitCode=0
Jul 16 22:24:25.145399 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:25.144996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerDied","Data":"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"}
Jul 16 22:24:26.151504 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:26.151469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerStarted","Data":"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"}
Jul 16 22:24:26.151981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:26.151590    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:24:26.178307 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:26.178245    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" podStartSLOduration=2.427957041 podStartE2EDuration="3.178229122s" totalImagesPullingTime="0s" totalInitContainerRuntime="750.272081ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:26.176106452 +0000 UTC m=+2066.271617783" watchObservedRunningTime="2026-07-16 22:24:26.178229122 +0000 UTC m=+2066.273740438"
Jul 16 22:24:57.158480 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:24:57.158443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:25:00.414361 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:25:00.414327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:25:00.418211 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:25:00.418189    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:27:19.912790 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:19.912747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:27:25.113864 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:25.113831    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:27:27.158263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:27.158216    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:27:31.407446 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:31.407404    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:27:34.934182 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:34.934130    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="main" containerID="cri-o://d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799" gracePeriod=45
Jul 16 22:27:35.188717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.188659    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:27:35.356883 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.356847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds\") pod \"bc1c5459-6acc-4c45-b257-cb9068fffa20\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") "
Jul 16 22:27:35.357128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.356908    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmx4c\" (UniqueName: \"kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c\") pod \"bc1c5459-6acc-4c45-b257-cb9068fffa20\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") "
Jul 16 22:27:35.357128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.356954    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location\") pod \"bc1c5459-6acc-4c45-b257-cb9068fffa20\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") "
Jul 16 22:27:35.357128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.356992    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs\") pod \"bc1c5459-6acc-4c45-b257-cb9068fffa20\" (UID: \"bc1c5459-6acc-4c45-b257-cb9068fffa20\") "
Jul 16 22:27:35.357316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.357128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "bc1c5459-6acc-4c45-b257-cb9068fffa20" (UID: "bc1c5459-6acc-4c45-b257-cb9068fffa20"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:35.357316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.357214    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:35.357681 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.357654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc1c5459-6acc-4c45-b257-cb9068fffa20" (UID: "bc1c5459-6acc-4c45-b257-cb9068fffa20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:35.359219 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.359198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c" (OuterVolumeSpecName: "kube-api-access-qmx4c") pod "bc1c5459-6acc-4c45-b257-cb9068fffa20" (UID: "bc1c5459-6acc-4c45-b257-cb9068fffa20"). InnerVolumeSpecName "kube-api-access-qmx4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:35.359309 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.359240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc1c5459-6acc-4c45-b257-cb9068fffa20" (UID: "bc1c5459-6acc-4c45-b257-cb9068fffa20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:35.458102 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.458071    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmx4c\" (UniqueName: \"kubernetes.io/projected/bc1c5459-6acc-4c45-b257-cb9068fffa20-kube-api-access-qmx4c\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:35.458102 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.458102    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc1c5459-6acc-4c45-b257-cb9068fffa20-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:35.458294 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:35.458112    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc1c5459-6acc-4c45-b257-cb9068fffa20-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:36.053809 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.053769    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerID="d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799" exitCode=0
Jul 16 22:27:36.054242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.053847    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"
Jul 16 22:27:36.054242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.053863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerDied","Data":"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"}
Jul 16 22:27:36.054242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.053926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q" event={"ID":"bc1c5459-6acc-4c45-b257-cb9068fffa20","Type":"ContainerDied","Data":"4712e5e53eaecc19a2ae4fa4020b514078b84764970bc5032c7420fb6baef60f"}
Jul 16 22:27:36.054242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.053952    2566 scope.go:117] "RemoveContainer" containerID="d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"
Jul 16 22:27:36.065306 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.065287    2566 scope.go:117] "RemoveContainer" containerID="838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"
Jul 16 22:27:36.074642 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.074623    2566 scope.go:117] "RemoveContainer" containerID="d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"
Jul 16 22:27:36.074970 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:27:36.074946    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799\": container with ID starting with d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799 not found: ID does not exist" containerID="d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"
Jul 16 22:27:36.075036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.074986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799"} err="failed to get container status \"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799\": rpc error: code = NotFound desc = could not find container \"d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799\": container with ID starting with d4ffe53341f26b3f9a5be02e8aaaaa89bfc0aa6d9fd3776622975fdaa3abd799 not found: ID does not exist"
Jul 16 22:27:36.075036 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.075015    2566 scope.go:117] "RemoveContainer" containerID="838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"
Jul 16 22:27:36.075485 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:27:36.075458    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816\": container with ID starting with 838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816 not found: ID does not exist" containerID="838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"
Jul 16 22:27:36.075590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.075494    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816"} err="failed to get container status \"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816\": rpc error: code = NotFound desc = could not find container \"838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816\": container with ID starting with 838011c39f85a4f8a7abe71409619528455a00043d6eb245a865cc18008d8816 not found: ID does not exist"
Jul 16 22:27:36.081839 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.081787    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:27:36.085192 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.085168    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-59cc6tcb6q"]
Jul 16 22:27:36.319929 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:36.319804    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" path="/var/lib/kubelet/pods/bc1c5459-6acc-4c45-b257-cb9068fffa20/volumes"
Jul 16 22:27:40.135335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.135265    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerName="main" containerID="cri-o://49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f" gracePeriod=45
Jul 16 22:27:40.392846 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.392758    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:27:40.500428 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.500390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds\") pod \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") "
Jul 16 22:27:40.500428 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.500428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs\") pod \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") "
Jul 16 22:27:40.500690 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.500504    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5wl8\" (UniqueName: \"kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8\") pod \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\" (UID: \"b2d18b99-f4bc-45f2-99e1-164e303fcbc4\") "
Jul 16 22:27:40.500751 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.500720    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b2d18b99-f4bc-45f2-99e1-164e303fcbc4" (UID: "b2d18b99-f4bc-45f2-99e1-164e303fcbc4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:40.502735 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.502706    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2d18b99-f4bc-45f2-99e1-164e303fcbc4" (UID: "b2d18b99-f4bc-45f2-99e1-164e303fcbc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:27:40.502871 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.502771    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8" (OuterVolumeSpecName: "kube-api-access-j5wl8") pod "b2d18b99-f4bc-45f2-99e1-164e303fcbc4" (UID: "b2d18b99-f4bc-45f2-99e1-164e303fcbc4"). InnerVolumeSpecName "kube-api-access-j5wl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:40.601535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.601493    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:40.601535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.601525    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:40.601535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:40.601534    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5wl8\" (UniqueName: \"kubernetes.io/projected/b2d18b99-f4bc-45f2-99e1-164e303fcbc4-kube-api-access-j5wl8\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:27:41.079111 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.079075    2566 generic.go:356] "Generic (PLEG): container finished" podID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerID="49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f" exitCode=0
Jul 16 22:27:41.079316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.079120    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"
Jul 16 22:27:41.079316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.079165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" event={"ID":"b2d18b99-f4bc-45f2-99e1-164e303fcbc4","Type":"ContainerDied","Data":"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"}
Jul 16 22:27:41.079316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.079226    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2" event={"ID":"b2d18b99-f4bc-45f2-99e1-164e303fcbc4","Type":"ContainerDied","Data":"5c60ae7bbed373014b81332d8bd27003ac13a38e7a1a3fbd67e03a323c68512a"}
Jul 16 22:27:41.079316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.079253    2566 scope.go:117] "RemoveContainer" containerID="49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"
Jul 16 22:27:41.088388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.088372    2566 scope.go:117] "RemoveContainer" containerID="49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"
Jul 16 22:27:41.088642 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:27:41.088624    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f\": container with ID starting with 49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f not found: ID does not exist" containerID="49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"
Jul 16 22:27:41.088695 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.088652    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f"} err="failed to get container status \"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f\": rpc error: code = NotFound desc = could not find container \"49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f\": container with ID starting with 49b32cf1f237818932a7602fc43294c17d927e1c43f584b8cfa00c4a806f316f not found: ID does not exist"
Jul 16 22:27:41.105605 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.105571    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:27:41.111766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:41.111741    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schestqb2"]
Jul 16 22:27:42.320325 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:42.320286    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" path="/var/lib/kubelet/pods/b2d18b99-f4bc-45f2-99e1-164e303fcbc4/volumes"
Jul 16 22:27:49.978156 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978124    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:27:49.978619 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978603    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="main"
Jul 16 22:27:49.978667 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978623    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="main"
Jul 16 22:27:49.978667 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978659    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="storage-initializer"
Jul 16 22:27:49.978731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978668    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="storage-initializer"
Jul 16 22:27:49.978731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978682    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerName="main"
Jul 16 22:27:49.978731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978694    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerName="main"
Jul 16 22:27:49.978895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978783    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1c5459-6acc-4c45-b257-cb9068fffa20" containerName="main"
Jul 16 22:27:49.978895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.978798    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d18b99-f4bc-45f2-99e1-164e303fcbc4" containerName="main"
Jul 16 22:27:49.983675 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.983655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:49.986586 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.986551    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:27:49.986691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.986613    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-qlxb7\""
Jul 16 22:27:49.986691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.986655    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:27:49.987673 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.987658    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 16 22:27:49.987755 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.987691    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:27:49.987950 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.987933    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:27:49.995154 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:49.995135    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:27:50.080237 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.080195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.080419 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.080256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.080419 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.080325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpct\" (UniqueName: \"kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.080419 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.080393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181161 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpct\" (UniqueName: \"kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181559 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.181634 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.181587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.183847 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.183809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.190137 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.190116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpct\" (UniqueName: \"kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct\") pod \"router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.294049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.293942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:50.431100 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:50.431072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:27:50.433043 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:27:50.433020    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a63e89c_f0ec_4221_882e_acd444ccd497.slice/crio-f38a92bb38de1c54b6faf3958252a354c9aae9c3f7d68c1ab35820080e255ee1 WatchSource:0}: Error finding container f38a92bb38de1c54b6faf3958252a354c9aae9c3f7d68c1ab35820080e255ee1: Status 404 returned error can't find the container with id f38a92bb38de1c54b6faf3958252a354c9aae9c3f7d68c1ab35820080e255ee1
Jul 16 22:27:51.126292 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:51.126255    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerStarted","Data":"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"}
Jul 16 22:27:51.126751 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:51.126315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerStarted","Data":"f38a92bb38de1c54b6faf3958252a354c9aae9c3f7d68c1ab35820080e255ee1"}
Jul 16 22:27:52.132668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:52.132635    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerID="ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8" exitCode=0
Jul 16 22:27:52.133097 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:52.132730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerDied","Data":"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"}
Jul 16 22:27:53.139240 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:53.139187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerStarted","Data":"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"}
Jul 16 22:27:53.139655 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:53.139292    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:27:53.164699 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:27:53.164645    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" podStartSLOduration=3.461581119 podStartE2EDuration="4.164624284s" totalImagesPullingTime="0s" totalInitContainerRuntime="703.043165ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:53.16402236 +0000 UTC m=+2273.259533677" watchObservedRunningTime="2026-07-16 22:27:53.164624284 +0000 UTC m=+2273.260135602"
Jul 16 22:28:10.430486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.430450    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:28:10.433994 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.433975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.437970 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.437945    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 22:28:10.438098 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.437950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-2tcm8\""
Jul 16 22:28:10.448985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.448953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:28:10.548968 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.548924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.549158 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.548989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.549158 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.549063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.549158 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.549138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qqh8\" (UniqueName: \"kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.649973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.649933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qqh8\" (UniqueName: \"kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.650169 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.649992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.650169 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.650047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.650169 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.650066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.650363 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.650343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.650400 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.650383    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.652674 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.652649    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.659504 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.659476    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qqh8\" (UniqueName: \"kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-549td\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.743343 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.743296    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:10.884506 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:10.884153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:28:10.886377 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:28:10.886348    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e828e3_0374_4a68_b9b8_9a10faeef650.slice/crio-f56792f1564705fa6dcea6c58c806f772289eacc5223ec1ecac7d434c98b83ec WatchSource:0}: Error finding container f56792f1564705fa6dcea6c58c806f772289eacc5223ec1ecac7d434c98b83ec: Status 404 returned error can't find the container with id f56792f1564705fa6dcea6c58c806f772289eacc5223ec1ecac7d434c98b83ec
Jul 16 22:28:11.228412 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:11.228377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerStarted","Data":"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"}
Jul 16 22:28:11.228579 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:11.228428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerStarted","Data":"f56792f1564705fa6dcea6c58c806f772289eacc5223ec1ecac7d434c98b83ec"}
Jul 16 22:28:12.235340 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:12.235303    2566 generic.go:356] "Generic (PLEG): container finished" podID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerID="ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a" exitCode=0
Jul 16 22:28:12.235734 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:12.235381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerDied","Data":"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"}
Jul 16 22:28:13.242071 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:13.242027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerStarted","Data":"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"}
Jul 16 22:28:13.242458 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:13.242128    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:28:13.271115 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:13.271053    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" podStartSLOduration=2.486935319 podStartE2EDuration="3.271036158s" totalImagesPullingTime="0s" totalInitContainerRuntime="784.100839ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:13.269546411 +0000 UTC m=+2293.365057738" watchObservedRunningTime="2026-07-16 22:28:13.271036158 +0000 UTC m=+2293.366547474"
Jul 16 22:28:24.146846 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:24.146794    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:28:44.248616 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:28:44.248589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:30:00.443730 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:30:00.443699    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:30:00.451392 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:30:00.451369    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:31:24.146797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:24.146759    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:31:24.232067 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:24.232035    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:31:31.760607 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:31.760572    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:31:34.146139 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:34.146100    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:31:34.248530 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:34.248493    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:31:39.253006 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.252965    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main" containerID="cri-o://86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011" gracePeriod=45
Jul 16 22:31:39.503537 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.503458    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:31:39.566650 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.566606    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs\") pod \"0a63e89c-f0ec-4221-882e-acd444ccd497\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") "
Jul 16 22:31:39.566873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.566696    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location\") pod \"0a63e89c-f0ec-4221-882e-acd444ccd497\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") "
Jul 16 22:31:39.566873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.566739    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgpct\" (UniqueName: \"kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct\") pod \"0a63e89c-f0ec-4221-882e-acd444ccd497\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") "
Jul 16 22:31:39.566873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.566763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds\") pod \"0a63e89c-f0ec-4221-882e-acd444ccd497\" (UID: \"0a63e89c-f0ec-4221-882e-acd444ccd497\") "
Jul 16 22:31:39.567124 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.567098    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0a63e89c-f0ec-4221-882e-acd444ccd497" (UID: "0a63e89c-f0ec-4221-882e-acd444ccd497"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:39.567393 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.567370    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a63e89c-f0ec-4221-882e-acd444ccd497" (UID: "0a63e89c-f0ec-4221-882e-acd444ccd497"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:39.569051 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.569019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a63e89c-f0ec-4221-882e-acd444ccd497" (UID: "0a63e89c-f0ec-4221-882e-acd444ccd497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:39.569161 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.569100    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct" (OuterVolumeSpecName: "kube-api-access-qgpct") pod "0a63e89c-f0ec-4221-882e-acd444ccd497" (UID: "0a63e89c-f0ec-4221-882e-acd444ccd497"). InnerVolumeSpecName "kube-api-access-qgpct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:39.668084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.668041    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a63e89c-f0ec-4221-882e-acd444ccd497-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:39.668084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.668075    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:39.668084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.668087    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpct\" (UniqueName: \"kubernetes.io/projected/0a63e89c-f0ec-4221-882e-acd444ccd497-kube-api-access-qgpct\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:39.668084 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:39.668096    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0a63e89c-f0ec-4221-882e-acd444ccd497-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:40.210162 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.210127    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerID="86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011" exitCode=0
Jul 16 22:31:40.210347 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.210182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"
Jul 16 22:31:40.210347 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.210217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerDied","Data":"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"}
Jul 16 22:31:40.210347 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.210274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb" event={"ID":"0a63e89c-f0ec-4221-882e-acd444ccd497","Type":"ContainerDied","Data":"f38a92bb38de1c54b6faf3958252a354c9aae9c3f7d68c1ab35820080e255ee1"}
Jul 16 22:31:40.210347 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.210294    2566 scope.go:117] "RemoveContainer" containerID="86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"
Jul 16 22:31:40.220082 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.220065    2566 scope.go:117] "RemoveContainer" containerID="ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"
Jul 16 22:31:40.228277 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.228259    2566 scope.go:117] "RemoveContainer" containerID="86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"
Jul 16 22:31:40.228551 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:31:40.228533    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011\": container with ID starting with 86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011 not found: ID does not exist" containerID="86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"
Jul 16 22:31:40.228618 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.228565    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011"} err="failed to get container status \"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011\": rpc error: code = NotFound desc = could not find container \"86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011\": container with ID starting with 86ba9df25d8d6f1c27c52b79c445bc550c743aa1c020f2f2f5168eb261681011 not found: ID does not exist"
Jul 16 22:31:40.228618 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.228591    2566 scope.go:117] "RemoveContainer" containerID="ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"
Jul 16 22:31:40.228862 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:31:40.228844    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8\": container with ID starting with ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8 not found: ID does not exist" containerID="ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"
Jul 16 22:31:40.228912 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.228870    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8"} err="failed to get container status \"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8\": rpc error: code = NotFound desc = could not find container \"ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8\": container with ID starting with ac15d60630bb3e509e9edb387e008e3cbee59c13b9f6eaa30d7ea152f1a817d8 not found: ID does not exist"
Jul 16 22:31:40.233676 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.233656    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:31:40.239452 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.239426    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-9f96557cc2rtjb"]
Jul 16 22:31:40.319637 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:40.319600    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" path="/var/lib/kubelet/pods/0a63e89c-f0ec-4221-882e-acd444ccd497/volumes"
Jul 16 22:31:44.248601 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:44.248565    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:31:45.608542 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.608509    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:31:45.609029 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.608986    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="storage-initializer"
Jul 16 22:31:45.609029 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.609006    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="storage-initializer"
Jul 16 22:31:45.609029 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.609027    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main"
Jul 16 22:31:45.609029 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.609032    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main"
Jul 16 22:31:45.609268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.609093    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a63e89c-f0ec-4221-882e-acd444ccd497" containerName="main"
Jul 16 22:31:45.614089 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.614070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.616686 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.616664    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-fx9mm\""
Jul 16 22:31:45.622270 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.622244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:31:45.721255 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.721225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfnj4\" (UniqueName: \"kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.721255 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.721262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.721519 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.721302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.721519 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.721387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.822345 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.822522 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.822522 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.822655 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfnj4\" (UniqueName: \"kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.822803 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.823016 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.822992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.825111 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.825090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.833309 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.833258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfnj4\" (UniqueName: \"kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4\") pod \"stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:45.924293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:45.924204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:46.082303 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:46.082275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:31:46.084176 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:31:46.084146    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a0e033_4856_4ced_9621_690051d53aef.slice/crio-af66a1c8df34db62f53c7495e2cd61d59a00af08648aec1fe73aeddbbbaf75cb WatchSource:0}: Error finding container af66a1c8df34db62f53c7495e2cd61d59a00af08648aec1fe73aeddbbbaf75cb: Status 404 returned error can't find the container with id af66a1c8df34db62f53c7495e2cd61d59a00af08648aec1fe73aeddbbbaf75cb
Jul 16 22:31:46.247267 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:46.247230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerStarted","Data":"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"}
Jul 16 22:31:46.247433 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:46.247280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerStarted","Data":"af66a1c8df34db62f53c7495e2cd61d59a00af08648aec1fe73aeddbbbaf75cb"}
Jul 16 22:31:46.781505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:46.781462    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main" containerID="cri-o://de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604" gracePeriod=45
Jul 16 22:31:47.046870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.046846    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:31:47.135142 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location\") pod \"33e828e3-0374-4a68-b9b8-9a10faeef650\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") "
Jul 16 22:31:47.135313 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds\") pod \"33e828e3-0374-4a68-b9b8-9a10faeef650\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") "
Jul 16 22:31:47.135313 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs\") pod \"33e828e3-0374-4a68-b9b8-9a10faeef650\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") "
Jul 16 22:31:47.135313 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qqh8\" (UniqueName: \"kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8\") pod \"33e828e3-0374-4a68-b9b8-9a10faeef650\" (UID: \"33e828e3-0374-4a68-b9b8-9a10faeef650\") "
Jul 16 22:31:47.135489 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135383    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "33e828e3-0374-4a68-b9b8-9a10faeef650" (UID: "33e828e3-0374-4a68-b9b8-9a10faeef650"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.135784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.135761    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33e828e3-0374-4a68-b9b8-9a10faeef650" (UID: "33e828e3-0374-4a68-b9b8-9a10faeef650"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:47.137416 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.137392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33e828e3-0374-4a68-b9b8-9a10faeef650" (UID: "33e828e3-0374-4a68-b9b8-9a10faeef650"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:31:47.137555 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.137533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8" (OuterVolumeSpecName: "kube-api-access-4qqh8") pod "33e828e3-0374-4a68-b9b8-9a10faeef650" (UID: "33e828e3-0374-4a68-b9b8-9a10faeef650"). InnerVolumeSpecName "kube-api-access-4qqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:47.236259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.236220    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qqh8\" (UniqueName: \"kubernetes.io/projected/33e828e3-0374-4a68-b9b8-9a10faeef650-kube-api-access-4qqh8\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:47.236259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.236251    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:47.236259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.236261    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33e828e3-0374-4a68-b9b8-9a10faeef650-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:47.236259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.236270    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e828e3-0374-4a68-b9b8-9a10faeef650-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:31:47.253662 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.253633    2566 generic.go:356] "Generic (PLEG): container finished" podID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerID="de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604" exitCode=0
Jul 16 22:31:47.253849 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.253684    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"
Jul 16 22:31:47.253849 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.253722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerDied","Data":"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"}
Jul 16 22:31:47.253849 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.253774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td" event={"ID":"33e828e3-0374-4a68-b9b8-9a10faeef650","Type":"ContainerDied","Data":"f56792f1564705fa6dcea6c58c806f772289eacc5223ec1ecac7d434c98b83ec"}
Jul 16 22:31:47.253849 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.253804    2566 scope.go:117] "RemoveContainer" containerID="de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"
Jul 16 22:31:47.255294 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.255269    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1a0e033-4856-4ced-9621-690051d53aef" containerID="70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74" exitCode=0
Jul 16 22:31:47.255396 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.255320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerDied","Data":"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"}
Jul 16 22:31:47.262946 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.262927    2566 scope.go:117] "RemoveContainer" containerID="ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"
Jul 16 22:31:47.271007 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.270974    2566 scope.go:117] "RemoveContainer" containerID="de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"
Jul 16 22:31:47.271256 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:31:47.271233    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604\": container with ID starting with de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604 not found: ID does not exist" containerID="de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"
Jul 16 22:31:47.271318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.271269    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604"} err="failed to get container status \"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604\": rpc error: code = NotFound desc = could not find container \"de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604\": container with ID starting with de7218051d3071fb062bb5e87ed21f083432b2a9a56843bb4b6dc6d0d1532604 not found: ID does not exist"
Jul 16 22:31:47.271318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.271292    2566 scope.go:117] "RemoveContainer" containerID="ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"
Jul 16 22:31:47.271547 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:31:47.271527    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a\": container with ID starting with ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a not found: ID does not exist" containerID="ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"
Jul 16 22:31:47.271638 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.271550    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a"} err="failed to get container status \"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a\": rpc error: code = NotFound desc = could not find container \"ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a\": container with ID starting with ac04d418797aca1ed885a93fed30009fcc32d8fdaf340ac5eef78ae9aa91881a not found: ID does not exist"
Jul 16 22:31:47.293066 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.293039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:31:47.298852 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:47.298828    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-549td"]
Jul 16 22:31:48.261981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:48.261941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerStarted","Data":"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"}
Jul 16 22:31:48.262379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:48.262072    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:31:48.283112 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:48.283055    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" podStartSLOduration=2.581030523 podStartE2EDuration="3.283037631s" totalImagesPullingTime="0s" totalInitContainerRuntime="702.007108ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:48.281231368 +0000 UTC m=+2508.376742686" watchObservedRunningTime="2026-07-16 22:31:48.283037631 +0000 UTC m=+2508.378548948"
Jul 16 22:31:48.319646 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:31:48.319612    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" path="/var/lib/kubelet/pods/33e828e3-0374-4a68-b9b8-9a10faeef650/volumes"
Jul 16 22:32:19.268048 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:32:19.268018    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:34:17.200880 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:17.200837    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:34:19.267976 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:19.267937    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:34:29.268179 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:29.268142    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:34:32.221770 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.221730    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main" containerID="cri-o://efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae" gracePeriod=45
Jul 16 22:34:32.481856 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.481767    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:34:32.567809 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.567768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs\") pod \"d1a0e033-4856-4ced-9621-690051d53aef\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") "
Jul 16 22:34:32.567809 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.567810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds\") pod \"d1a0e033-4856-4ced-9621-690051d53aef\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") "
Jul 16 22:34:32.568073 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.567872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfnj4\" (UniqueName: \"kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4\") pod \"d1a0e033-4856-4ced-9621-690051d53aef\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") "
Jul 16 22:34:32.568073 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.567910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location\") pod \"d1a0e033-4856-4ced-9621-690051d53aef\" (UID: \"d1a0e033-4856-4ced-9621-690051d53aef\") "
Jul 16 22:34:32.568170 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.568129    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d1a0e033-4856-4ced-9621-690051d53aef" (UID: "d1a0e033-4856-4ced-9621-690051d53aef"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:32.568645 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.568626    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1a0e033-4856-4ced-9621-690051d53aef" (UID: "d1a0e033-4856-4ced-9621-690051d53aef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:34:32.570194 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.570167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4" (OuterVolumeSpecName: "kube-api-access-tfnj4") pod "d1a0e033-4856-4ced-9621-690051d53aef" (UID: "d1a0e033-4856-4ced-9621-690051d53aef"). InnerVolumeSpecName "kube-api-access-tfnj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:34:32.570304 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.570196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1a0e033-4856-4ced-9621-690051d53aef" (UID: "d1a0e033-4856-4ced-9621-690051d53aef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:34:32.669250 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.669212    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1a0e033-4856-4ced-9621-690051d53aef-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:34:32.669250 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.669244    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:34:32.669250 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.669255    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfnj4\" (UniqueName: \"kubernetes.io/projected/d1a0e033-4856-4ced-9621-690051d53aef-kube-api-access-tfnj4\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:34:32.669494 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:32.669264    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1a0e033-4856-4ced-9621-690051d53aef-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:34:33.038755 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.038717    2566 generic.go:356] "Generic (PLEG): container finished" podID="d1a0e033-4856-4ced-9621-690051d53aef" containerID="efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae" exitCode=0
Jul 16 22:34:33.038966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.038765    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"
Jul 16 22:34:33.038966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.038799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerDied","Data":"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"}
Jul 16 22:34:33.038966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.038873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg" event={"ID":"d1a0e033-4856-4ced-9621-690051d53aef","Type":"ContainerDied","Data":"af66a1c8df34db62f53c7495e2cd61d59a00af08648aec1fe73aeddbbbaf75cb"}
Jul 16 22:34:33.038966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.038895    2566 scope.go:117] "RemoveContainer" containerID="efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"
Jul 16 22:34:33.048059 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.048039    2566 scope.go:117] "RemoveContainer" containerID="70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"
Jul 16 22:34:33.056026 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.056005    2566 scope.go:117] "RemoveContainer" containerID="efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"
Jul 16 22:34:33.056328 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:34:33.056309    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae\": container with ID starting with efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae not found: ID does not exist" containerID="efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"
Jul 16 22:34:33.056372 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.056339    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae"} err="failed to get container status \"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae\": rpc error: code = NotFound desc = could not find container \"efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae\": container with ID starting with efd791535c7ed75d41ac5db7b239d426603ccda6ba707847ca67d7e701b53fae not found: ID does not exist"
Jul 16 22:34:33.056372 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.056357    2566 scope.go:117] "RemoveContainer" containerID="70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"
Jul 16 22:34:33.056603 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:34:33.056584    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74\": container with ID starting with 70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74 not found: ID does not exist" containerID="70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"
Jul 16 22:34:33.056661 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.056607    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74"} err="failed to get container status \"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74\": rpc error: code = NotFound desc = could not find container \"70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74\": container with ID starting with 70bcae0f2b3de45a74caeed40d35b79a1dd8113ce7076ed901909d3aa02bdb74 not found: ID does not exist"
Jul 16 22:34:33.061136 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.061105    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:34:33.063479 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:33.063455    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-59cf7696fd-54khg"]
Jul 16 22:34:34.319938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:34:34.319905    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1a0e033-4856-4ced-9621-690051d53aef" path="/var/lib/kubelet/pods/d1a0e033-4856-4ced-9621-690051d53aef/volumes"
Jul 16 22:35:00.471855 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:00.471758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:35:00.477470 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:00.477449    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:35:12.867767 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.867728    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868077    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="storage-initializer"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868089    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="storage-initializer"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868112    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="storage-initializer"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868117    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="storage-initializer"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868124    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868134    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main"
Jul 16 22:35:12.868168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868139    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main"
Jul 16 22:35:12.868426 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868207    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a0e033-4856-4ced-9621-690051d53aef" containerName="main"
Jul 16 22:35:12.868426 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.868215    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33e828e3-0374-4a68-b9b8-9a10faeef650" containerName="main"
Jul 16 22:35:12.871323 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.871295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.874229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.874200    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:35:12.874229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.874221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:35:12.875313 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.875290    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:35:12.875313 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.875306    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-2fllx\""
Jul 16 22:35:12.875507 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.875335    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 16 22:35:12.875507 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.875298    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:35:12.885277 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.885242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:35:12.914611 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914611 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914866 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvdc\" (UniqueName: \"kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914866 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914866 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:12.914973 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:12.914906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.015775 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.015981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.015981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvdc\" (UniqueName: \"kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.015981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.015981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.015989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.016097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016296 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.016221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016393 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.016368    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016465 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.016446    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.016513 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.016480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.018293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.018266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.018430 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.018412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.032707 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.032674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvdc\" (UniqueName: \"kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.182614 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.182519    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:35:13.275438 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.275398    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:35:13.279120 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.279094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.284972 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.284946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-nwvj4\""
Jul 16 22:35:13.304441 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.304405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:35:13.319427 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.319392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.319701 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.319475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.319701 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.319518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vghjb\" (UniqueName: \"kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.319701 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.319657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.381642 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.381615    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:35:13.383804 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:35:13.383776    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c571617_70c8_4740_9bd2_b66101a09672.slice/crio-a77ea6c24f630d6351d18d1ced23a15ab34fc7f824757618a2a7f492344b1e1f WatchSource:0}: Error finding container a77ea6c24f630d6351d18d1ced23a15ab34fc7f824757618a2a7f492344b1e1f: Status 404 returned error can't find the container with id a77ea6c24f630d6351d18d1ced23a15ab34fc7f824757618a2a7f492344b1e1f
Jul 16 22:35:13.420373 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.420555 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.420634 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.420702 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vghjb\" (UniqueName: \"kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.420845 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.420928 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.420795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.424126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.424097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.460424 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.460393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vghjb\" (UniqueName: \"kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.590717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.590682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:13.763185 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:13.763160    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:35:13.765617 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:35:13.765596    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f8cb3_3429_4d11_8516_3ae2dc646be0.slice/crio-64af6ebb9c9002bd54c6b08c272f171bb7aaa07c5c7ae0cb10f0253b568bdbd0 WatchSource:0}: Error finding container 64af6ebb9c9002bd54c6b08c272f171bb7aaa07c5c7ae0cb10f0253b568bdbd0: Status 404 returned error can't find the container with id 64af6ebb9c9002bd54c6b08c272f171bb7aaa07c5c7ae0cb10f0253b568bdbd0
Jul 16 22:35:14.231787 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:14.231750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerStarted","Data":"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"}
Jul 16 22:35:14.232245 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:14.231865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerStarted","Data":"64af6ebb9c9002bd54c6b08c272f171bb7aaa07c5c7ae0cb10f0253b568bdbd0"}
Jul 16 22:35:14.233475 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:14.233447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerStarted","Data":"20207683626b75752decf1f1d3868f1433b4c7dafa22d44d33d2fc916adefe15"}
Jul 16 22:35:14.233585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:14.233498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerStarted","Data":"a77ea6c24f630d6351d18d1ced23a15ab34fc7f824757618a2a7f492344b1e1f"}
Jul 16 22:35:15.242620 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:15.242581    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerID="fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9" exitCode=0
Jul 16 22:35:15.243147 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:15.242690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerDied","Data":"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"}
Jul 16 22:35:16.251418 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:16.251382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerStarted","Data":"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"}
Jul 16 22:35:16.251919 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:16.251493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:16.277129 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:16.277055    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" podStartSLOduration=2.279854648 podStartE2EDuration="3.277031009s" totalImagesPullingTime="0s" totalInitContainerRuntime="997.176361ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:16.274023766 +0000 UTC m=+2716.369535117" watchObservedRunningTime="2026-07-16 22:35:16.277031009 +0000 UTC m=+2716.372542328"
Jul 16 22:35:17.257236 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:17.257200    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c571617-70c8-4740-9bd2-b66101a09672" containerID="20207683626b75752decf1f1d3868f1433b4c7dafa22d44d33d2fc916adefe15" exitCode=0
Jul 16 22:35:17.257600 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:17.257273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerDied","Data":"20207683626b75752decf1f1d3868f1433b4c7dafa22d44d33d2fc916adefe15"}
Jul 16 22:35:17.258670 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:17.258650    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:35:47.259698 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:47.259206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:35:57.475268 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:35:57.475174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerStarted","Data":"dbe53e3ba00a7f5aeb8affa1d620054a503e09c87ef288f7ce2ac28f32944395"}
Jul 16 22:37:35.631535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:35.631460    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=100.480168765 podStartE2EDuration="2m23.631439092s" totalImagesPullingTime="39.96116883s" totalInitContainerRuntime="3.190101497s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:57.504137839 +0000 UTC m=+2757.599649156" watchObservedRunningTime="2026-07-16 22:37:35.631439092 +0000 UTC m=+2855.726950411"
Jul 16 22:37:35.633851 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:35.633808    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:37:37.259347 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:37.259307    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:37:47.259197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:47.259157    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:37:50.654482 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.654361    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main" containerID="cri-o://a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377" gracePeriod=45
Jul 16 22:37:50.910549 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.910495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:37:50.912676 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.912656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs\") pod \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") "
Jul 16 22:37:50.912797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.912692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vghjb\" (UniqueName: \"kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb\") pod \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") "
Jul 16 22:37:50.912902 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.912879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds\") pod \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") "
Jul 16 22:37:50.912964 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.912925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location\") pod \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\" (UID: \"0b1f8cb3-3429-4d11-8516-3ae2dc646be0\") "
Jul 16 22:37:50.913099 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.913074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "0b1f8cb3-3429-4d11-8516-3ae2dc646be0" (UID: "0b1f8cb3-3429-4d11-8516-3ae2dc646be0"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:50.913209 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.913187    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:37:50.913613 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.913589    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b1f8cb3-3429-4d11-8516-3ae2dc646be0" (UID: "0b1f8cb3-3429-4d11-8516-3ae2dc646be0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:37:50.914740 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.914719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b1f8cb3-3429-4d11-8516-3ae2dc646be0" (UID: "0b1f8cb3-3429-4d11-8516-3ae2dc646be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:37:50.915090 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:50.915068    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb" (OuterVolumeSpecName: "kube-api-access-vghjb") pod "0b1f8cb3-3429-4d11-8516-3ae2dc646be0" (UID: "0b1f8cb3-3429-4d11-8516-3ae2dc646be0"). InnerVolumeSpecName "kube-api-access-vghjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:37:51.013590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.013560    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.013590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.013588    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vghjb\" (UniqueName: \"kubernetes.io/projected/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kube-api-access-vghjb\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.013746 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.013597    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b1f8cb3-3429-4d11-8516-3ae2dc646be0-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:37:51.053958 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.053923    2566 generic.go:356] "Generic (PLEG): container finished" podID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerID="a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377" exitCode=0
Jul 16 22:37:51.054077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.053976    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"
Jul 16 22:37:51.054077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.054019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerDied","Data":"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"}
Jul 16 22:37:51.054077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.054064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt" event={"ID":"0b1f8cb3-3429-4d11-8516-3ae2dc646be0","Type":"ContainerDied","Data":"64af6ebb9c9002bd54c6b08c272f171bb7aaa07c5c7ae0cb10f0253b568bdbd0"}
Jul 16 22:37:51.054181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.054084    2566 scope.go:117] "RemoveContainer" containerID="a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"
Jul 16 22:37:51.066906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.066888    2566 scope.go:117] "RemoveContainer" containerID="fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"
Jul 16 22:37:51.076050 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.076033    2566 scope.go:117] "RemoveContainer" containerID="a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"
Jul 16 22:37:51.076301 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:37:51.076275    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377\": container with ID starting with a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377 not found: ID does not exist" containerID="a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"
Jul 16 22:37:51.076411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.076310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377"} err="failed to get container status \"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377\": rpc error: code = NotFound desc = could not find container \"a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377\": container with ID starting with a76da18b394aacc5bd1e6016d6dc30184fb4a2e8c3a37b15790145ef8e807377 not found: ID does not exist"
Jul 16 22:37:51.076411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.076333    2566 scope.go:117] "RemoveContainer" containerID="fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"
Jul 16 22:37:51.076571 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:37:51.076549    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9\": container with ID starting with fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9 not found: ID does not exist" containerID="fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"
Jul 16 22:37:51.076629 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.076581    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9"} err="failed to get container status \"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9\": rpc error: code = NotFound desc = could not find container \"fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9\": container with ID starting with fda5b31911b607dc4215ef010e0e65aa380640b7d901e04128da5863d96913a9 not found: ID does not exist"
Jul 16 22:37:51.079971 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.079951    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:37:51.084252 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:51.084234    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche5fptt"]
Jul 16 22:37:52.319197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:52.319162    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" path="/var/lib/kubelet/pods/0b1f8cb3-3429-4d11-8516-3ae2dc646be0/volumes"
Jul 16 22:37:52.454870 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:37:52.454831    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:38:07.478641 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:07.478598    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="main" containerID="cri-o://dbe53e3ba00a7f5aeb8affa1d620054a503e09c87ef288f7ce2ac28f32944395" gracePeriod=45
Jul 16 22:38:08.135434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.135396    2566 generic.go:356] "Generic (PLEG): container finished" podID="2c571617-70c8-4740-9bd2-b66101a09672" containerID="dbe53e3ba00a7f5aeb8affa1d620054a503e09c87ef288f7ce2ac28f32944395" exitCode=0
Jul 16 22:38:08.135603 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.135461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerDied","Data":"dbe53e3ba00a7f5aeb8affa1d620054a503e09c87ef288f7ce2ac28f32944395"}
Jul 16 22:38:08.231329 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.231304    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:38:08.361042 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.360925    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361042 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.360985    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361042 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361009    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361341 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361341 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpvdc\" (UniqueName: \"kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361341 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361341 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361198    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home\") pod \"2c571617-70c8-4740-9bd2-b66101a09672\" (UID: \"2c571617-70c8-4740-9bd2-b66101a09672\") "
Jul 16 22:38:08.361341 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:38:08.361602 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.361543    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.362139 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.362106    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home" (OuterVolumeSpecName: "home") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:38:08.363944 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.363916    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm" (OuterVolumeSpecName: "dshm") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:38:08.364047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.363960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:38:08.364096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.364074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc" (OuterVolumeSpecName: "kube-api-access-jpvdc") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "kube-api-access-jpvdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:38:08.373130 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.373102    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:38:08.421324 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.421244    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c571617-70c8-4740-9bd2-b66101a09672" (UID: "2c571617-70c8-4740-9bd2-b66101a09672"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:38:08.462867 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462808    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvdc\" (UniqueName: \"kubernetes.io/projected/2c571617-70c8-4740-9bd2-b66101a09672-kube-api-access-jpvdc\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.462867 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462866    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.463072 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462877    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.463072 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462885    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.463072 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462893    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c571617-70c8-4740-9bd2-b66101a09672-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:08.463072 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:08.462915    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c571617-70c8-4740-9bd2-b66101a09672-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:38:09.142785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.142742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2c571617-70c8-4740-9bd2-b66101a09672","Type":"ContainerDied","Data":"a77ea6c24f630d6351d18d1ced23a15ab34fc7f824757618a2a7f492344b1e1f"}
Jul 16 22:38:09.143218 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.142765    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 22:38:09.143218 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.142845    2566 scope.go:117] "RemoveContainer" containerID="dbe53e3ba00a7f5aeb8affa1d620054a503e09c87ef288f7ce2ac28f32944395"
Jul 16 22:38:09.154483 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.154457    2566 scope.go:117] "RemoveContainer" containerID="20207683626b75752decf1f1d3868f1433b4c7dafa22d44d33d2fc916adefe15"
Jul 16 22:38:09.168529 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.168501    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:38:09.172935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:09.172910    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 22:38:10.320951 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:10.320915    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c571617-70c8-4740-9bd2-b66101a09672" path="/var/lib/kubelet/pods/2c571617-70c8-4740-9bd2-b66101a09672/volumes"
Jul 16 22:38:35.094344 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094307    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:38:35.094812 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094776    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="main"
Jul 16 22:38:35.094812 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094791    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="main"
Jul 16 22:38:35.094812 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094804    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094840    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="storage-initializer"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094846    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="storage-initializer"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094854    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="storage-initializer"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094859    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="storage-initializer"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094914    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1f8cb3-3429-4d11-8516-3ae2dc646be0" containerName="main"
Jul 16 22:38:35.094935 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.094924    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c571617-70c8-4740-9bd2-b66101a09672" containerName="main"
Jul 16 22:38:35.099507 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.099490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.103169 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.103143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:38:35.103310 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.103203    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:38:35.103310 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.103210    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:38:35.103310 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.103249    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 16 22:38:35.108288 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.108262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:38:35.195517 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.195691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.195691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.195691 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195599    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.195801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwns\" (UniqueName: \"kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.195801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.195723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.296906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.296863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.296906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.296908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.296969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwns\" (UniqueName: \"kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297126 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297361 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297401 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297376    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.297446 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.297408    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.299417 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.299390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.299713 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.299698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.306041 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.306015    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwns\" (UniqueName: \"kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.410441 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.410339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:35.551353 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.551324    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:38:35.555772 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.555745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.558575 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.558554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:38:35.559123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.559103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-4q2zl\""
Jul 16 22:38:35.561291 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:38:35.561270    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec58e32_a781_4248_92a1_451b569f969a.slice/crio-fbf0ef5d80e4a93a9d1cde240cba157fc8f844cf155fdfb98a128e13e8689c2d WatchSource:0}: Error finding container fbf0ef5d80e4a93a9d1cde240cba157fc8f844cf155fdfb98a128e13e8689c2d: Status 404 returned error can't find the container with id fbf0ef5d80e4a93a9d1cde240cba157fc8f844cf155fdfb98a128e13e8689c2d
Jul 16 22:38:35.569078 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.569055    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:38:35.701379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.701295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.701379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.701339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9q9m\" (UniqueName: \"kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.701576 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.701472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.802301 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.802265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.802505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.802326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.802505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.802352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9q9m\" (UniqueName: \"kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.802763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.802743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.804944 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.804921    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.812885 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.812861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9q9m\" (UniqueName: \"kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m\") pod \"scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:35.875468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:35.875424    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:36.014535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.014509    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:38:36.016578 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:38:36.016549    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e5b282_5141_42a7_bfb4_10ac81363a73.slice/crio-ca428cded55ed6dbe12e4b67a715651f03adc3db46000423ed25f1434fe5c06f WatchSource:0}: Error finding container ca428cded55ed6dbe12e4b67a715651f03adc3db46000423ed25f1434fe5c06f: Status 404 returned error can't find the container with id ca428cded55ed6dbe12e4b67a715651f03adc3db46000423ed25f1434fe5c06f
Jul 16 22:38:36.269787 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.269693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" event={"ID":"76e5b282-5141-42a7-bfb4-10ac81363a73","Type":"ContainerStarted","Data":"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"}
Jul 16 22:38:36.269787 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.269742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" event={"ID":"76e5b282-5141-42a7-bfb4-10ac81363a73","Type":"ContainerStarted","Data":"ca428cded55ed6dbe12e4b67a715651f03adc3db46000423ed25f1434fe5c06f"}
Jul 16 22:38:36.270289 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.269790    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:38:36.271398 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.271378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" event={"ID":"fec58e32-a781-4248-92a1-451b569f969a","Type":"ContainerStarted","Data":"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"}
Jul 16 22:38:36.271499 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.271413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" event={"ID":"fec58e32-a781-4248-92a1-451b569f969a","Type":"ContainerStarted","Data":"fbf0ef5d80e4a93a9d1cde240cba157fc8f844cf155fdfb98a128e13e8689c2d"}
Jul 16 22:38:36.301605 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.301548    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" podStartSLOduration=1.301532776 podStartE2EDuration="1.301532776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:36.29950773 +0000 UTC m=+2916.395019047" watchObservedRunningTime="2026-07-16 22:38:36.301532776 +0000 UTC m=+2916.397044093"
Jul 16 22:38:36.410591 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:36.410560    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:45.411486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:45.411448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:45.423987 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:45.423963    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:38:45.442652 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:45.442605    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" podStartSLOduration=10.442590184 podStartE2EDuration="10.442590184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:36.374209501 +0000 UTC m=+2916.469720817" watchObservedRunningTime="2026-07-16 22:38:45.442590184 +0000 UTC m=+2925.538101502"
Jul 16 22:38:46.333293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:38:46.333267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:39:07.278265 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:07.278236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:39:08.777608 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:08.777576    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:39:08.787971 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:08.787935    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:39:17.278507 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:17.278474    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:39:23.798776 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:23.798671    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" podUID="fec58e32-a781-4248-92a1-451b569f969a" containerName="main" containerID="cri-o://0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac" gracePeriod=45
Jul 16 22:39:23.812501 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:23.812468    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerName="main" containerID="cri-o://8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16" gracePeriod=45
Jul 16 22:39:24.084804 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.084779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:39:24.087785 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.087763    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:39:24.202398 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.202398 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202398    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.202668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.202668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9q9m\" (UniqueName: \"kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m\") pod \"76e5b282-5141-42a7-bfb4-10ac81363a73\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") "
Jul 16 22:39:24.202668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds\") pod \"76e5b282-5141-42a7-bfb4-10ac81363a73\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") "
Jul 16 22:39:24.202668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhwns\" (UniqueName: \"kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.202668 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.202977 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs\") pod \"76e5b282-5141-42a7-bfb4-10ac81363a73\" (UID: \"76e5b282-5141-42a7-bfb4-10ac81363a73\") "
Jul 16 22:39:24.202977 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home" (OuterVolumeSpecName: "home") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:24.202977 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:24.202977 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.202795    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache\") pod \"fec58e32-a781-4248-92a1-451b569f969a\" (UID: \"fec58e32-a781-4248-92a1-451b569f969a\") "
Jul 16 22:39:24.203190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.203128    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:24.203190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.203154    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.203190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.203170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "76e5b282-5141-42a7-bfb4-10ac81363a73" (UID: "76e5b282-5141-42a7-bfb4-10ac81363a73"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:24.203190 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.203178    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.204880 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.204850    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:39:24.204985 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.204944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm" (OuterVolumeSpecName: "dshm") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:24.205090 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.205072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76e5b282-5141-42a7-bfb4-10ac81363a73" (UID: "76e5b282-5141-42a7-bfb4-10ac81363a73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:39:24.205411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.205387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m" (OuterVolumeSpecName: "kube-api-access-d9q9m") pod "76e5b282-5141-42a7-bfb4-10ac81363a73" (UID: "76e5b282-5141-42a7-bfb4-10ac81363a73"). InnerVolumeSpecName "kube-api-access-d9q9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:24.205411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.205406    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns" (OuterVolumeSpecName: "kube-api-access-vhwns") pod "fec58e32-a781-4248-92a1-451b569f969a" (UID: "fec58e32-a781-4248-92a1-451b569f969a"). InnerVolumeSpecName "kube-api-access-vhwns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:24.304379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304344    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/76e5b282-5141-42a7-bfb4-10ac81363a73-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304372    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwns\" (UniqueName: \"kubernetes.io/projected/fec58e32-a781-4248-92a1-451b569f969a-kube-api-access-vhwns\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304382    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fec58e32-a781-4248-92a1-451b569f969a-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304379 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304394    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e5b282-5141-42a7-bfb4-10ac81363a73-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304653 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304404    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304653 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304411    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fec58e32-a781-4248-92a1-451b569f969a-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.304653 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.304419    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9q9m\" (UniqueName: \"kubernetes.io/projected/76e5b282-5141-42a7-bfb4-10ac81363a73-kube-api-access-d9q9m\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:39:24.495758 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.495720    2566 generic.go:356] "Generic (PLEG): container finished" podID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerID="8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16" exitCode=0
Jul 16 22:39:24.495950 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.495785    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"
Jul 16 22:39:24.495950 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.495799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" event={"ID":"76e5b282-5141-42a7-bfb4-10ac81363a73","Type":"ContainerDied","Data":"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"}
Jul 16 22:39:24.495950 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.495912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws" event={"ID":"76e5b282-5141-42a7-bfb4-10ac81363a73","Type":"ContainerDied","Data":"ca428cded55ed6dbe12e4b67a715651f03adc3db46000423ed25f1434fe5c06f"}
Jul 16 22:39:24.495950 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.495944    2566 scope.go:117] "RemoveContainer" containerID="8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"
Jul 16 22:39:24.497196 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.497176    2566 generic.go:356] "Generic (PLEG): container finished" podID="fec58e32-a781-4248-92a1-451b569f969a" containerID="0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac" exitCode=0
Jul 16 22:39:24.497300 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.497205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" event={"ID":"fec58e32-a781-4248-92a1-451b569f969a","Type":"ContainerDied","Data":"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"}
Jul 16 22:39:24.497300 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.497237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"
Jul 16 22:39:24.497300 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.497259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf" event={"ID":"fec58e32-a781-4248-92a1-451b569f969a","Type":"ContainerDied","Data":"fbf0ef5d80e4a93a9d1cde240cba157fc8f844cf155fdfb98a128e13e8689c2d"}
Jul 16 22:39:24.505062 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.505041    2566 scope.go:117] "RemoveContainer" containerID="8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"
Jul 16 22:39:24.505361 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:39:24.505341    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16\": container with ID starting with 8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16 not found: ID does not exist" containerID="8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"
Jul 16 22:39:24.505439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.505375    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16"} err="failed to get container status \"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16\": rpc error: code = NotFound desc = could not find container \"8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16\": container with ID starting with 8eccfd626127c61945bdb374d384ca9e6b1124eedaebf40bec8864441fcbeb16 not found: ID does not exist"
Jul 16 22:39:24.505439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.505399    2566 scope.go:117] "RemoveContainer" containerID="0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"
Jul 16 22:39:24.513708 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.513688    2566 scope.go:117] "RemoveContainer" containerID="0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"
Jul 16 22:39:24.514013 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:39:24.513995    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac\": container with ID starting with 0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac not found: ID does not exist" containerID="0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"
Jul 16 22:39:24.514093 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.514020    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac"} err="failed to get container status \"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac\": rpc error: code = NotFound desc = could not find container \"0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac\": container with ID starting with 0b7fa59446baa9d15b4f78bb6ea3ee87f3041ecb542823516deb8140c0c118ac not found: ID does not exist"
Jul 16 22:39:24.514735 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.514715    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:39:24.523294 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.523268    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-dqvgf"]
Jul 16 22:39:24.536537 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.536508    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:39:24.549536 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:24.549510    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-568fcc29ws"]
Jul 16 22:39:26.321185 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:26.321151    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" path="/var/lib/kubelet/pods/76e5b282-5141-42a7-bfb4-10ac81363a73/volumes"
Jul 16 22:39:26.321545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:26.321513    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec58e32-a781-4248-92a1-451b569f969a" path="/var/lib/kubelet/pods/fec58e32-a781-4248-92a1-451b569f969a/volumes"
Jul 16 22:39:37.010308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010266    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:39:37.010797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010749    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec58e32-a781-4248-92a1-451b569f969a" containerName="main"
Jul 16 22:39:37.010797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010767    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec58e32-a781-4248-92a1-451b569f969a" containerName="main"
Jul 16 22:39:37.010797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerName="main"
Jul 16 22:39:37.010797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010784    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerName="main"
Jul 16 22:39:37.011040 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e5b282-5141-42a7-bfb4-10ac81363a73" containerName="main"
Jul 16 22:39:37.011040 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.010960    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec58e32-a781-4248-92a1-451b569f969a" containerName="main"
Jul 16 22:39:37.015404 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.015383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.019581 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.019560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 16 22:39:37.019581 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.019579    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:39:37.019784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.019622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:39:37.019784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.019681    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:39:37.026613 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.026591    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:39:37.115471 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.115471 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.115712 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257l4\" (UniqueName: \"kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.115712 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.115712 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.115712 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.115660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217029 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.216990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257l4\" (UniqueName: \"kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217511 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217486    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217567 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.217622 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.217563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.219538 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.219515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.219799 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.219780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.226914 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.226881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-257l4\" (UniqueName: \"kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.326754 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.326670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:37.480201 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.480163    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:39:37.484631 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.484608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.488854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.488827    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-2rtx2\""
Jul 16 22:39:37.491284 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.491254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:39:37.493415 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:39:37.493392    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835aa9df_efd4_42b2_af58_6a781edbc921.slice/crio-2207eb1258343da99be4347dbd4b92ab08b621dc1543bf8f154f0109f65f4463 WatchSource:0}: Error finding container 2207eb1258343da99be4347dbd4b92ab08b621dc1543bf8f154f0109f65f4463: Status 404 returned error can't find the container with id 2207eb1258343da99be4347dbd4b92ab08b621dc1543bf8f154f0109f65f4463
Jul 16 22:39:37.500676 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.500653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:39:37.576856 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.576741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" event={"ID":"835aa9df-efd4-42b2-af58-6a781edbc921","Type":"ContainerStarted","Data":"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"}
Jul 16 22:39:37.577059 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.576853    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" event={"ID":"835aa9df-efd4-42b2-af58-6a781edbc921","Type":"ContainerStarted","Data":"2207eb1258343da99be4347dbd4b92ab08b621dc1543bf8f154f0109f65f4463"}
Jul 16 22:39:37.605841 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.605619    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" podStartSLOduration=1.605597411 podStartE2EDuration="1.605597411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:37.605477419 +0000 UTC m=+2977.700988737" watchObservedRunningTime="2026-07-16 22:39:37.605597411 +0000 UTC m=+2977.701108726"
Jul 16 22:39:37.620903 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.620871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.621077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.620942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.621077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.620994    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvbxg\" (UniqueName: \"kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.722395 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.722341    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.722616 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.722403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvbxg\" (UniqueName: \"kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.722616 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.722444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.722781 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.722749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.725308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.725277    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.742549 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.742524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvbxg\" (UniqueName: \"kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.804526 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.804494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:37.952306 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:37.952277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:39:37.954457 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:39:37.954421    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bfccf4_561c_4694_ba9e_005f3488daf7.slice/crio-e89c2329f0ef44032343cafb553af3b839a81fa6f784517249f0b37d5d24ad91 WatchSource:0}: Error finding container e89c2329f0ef44032343cafb553af3b839a81fa6f784517249f0b37d5d24ad91: Status 404 returned error can't find the container with id e89c2329f0ef44032343cafb553af3b839a81fa6f784517249f0b37d5d24ad91
Jul 16 22:39:38.327054 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:38.327024    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:38.583473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:38.583381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" event={"ID":"30bfccf4-561c-4694-ba9e-005f3488daf7","Type":"ContainerStarted","Data":"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"}
Jul 16 22:39:38.583473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:38.583428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" event={"ID":"30bfccf4-561c-4694-ba9e-005f3488daf7","Type":"ContainerStarted","Data":"e89c2329f0ef44032343cafb553af3b839a81fa6f784517249f0b37d5d24ad91"}
Jul 16 22:39:38.583473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:38.583458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:39:38.606566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:38.606511    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" podStartSLOduration=1.606493658 podStartE2EDuration="1.606493658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:38.604305733 +0000 UTC m=+2978.699817050" watchObservedRunningTime="2026-07-16 22:39:38.606493658 +0000 UTC m=+2978.702004974"
Jul 16 22:39:47.327001 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:47.326963    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:47.339865 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:47.339841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:39:47.636583 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:39:47.636499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:40:00.505444 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:00.505418    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:40:00.513496 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:00.513467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:40:09.589854 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:09.589795    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:40:10.863572 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:10.863531    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:40:10.865446 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:10.865412    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:40:19.589471 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:19.589419    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:40:25.884742 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:25.884677    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerName="main" containerID="cri-o://16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59" gracePeriod=45
Jul 16 22:40:25.885559 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:25.885528    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" podUID="835aa9df-efd4-42b2-af58-6a781edbc921" containerName="main" containerID="cri-o://9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f" gracePeriod=45
Jul 16 22:40:26.164536 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.164510    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:40:26.167178 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.167158    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:40:26.248207 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.248388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248215    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.248388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs\") pod \"30bfccf4-561c-4694-ba9e-005f3488daf7\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") "
Jul 16 22:40:26.248388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.248388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248381    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvbxg\" (UniqueName: \"kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg\") pod \"30bfccf4-561c-4694-ba9e-005f3488daf7\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") "
Jul 16 22:40:26.248633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248422    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds\") pod \"30bfccf4-561c-4694-ba9e-005f3488daf7\" (UID: \"30bfccf4-561c-4694-ba9e-005f3488daf7\") "
Jul 16 22:40:26.248633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248449    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.248633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.248633 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.248537    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-257l4\" (UniqueName: \"kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4\") pod \"835aa9df-efd4-42b2-af58-6a781edbc921\" (UID: \"835aa9df-efd4-42b2-af58-6a781edbc921\") "
Jul 16 22:40:26.249193 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.249061    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "30bfccf4-561c-4694-ba9e-005f3488daf7" (UID: "30bfccf4-561c-4694-ba9e-005f3488daf7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:26.249193 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.249165    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home" (OuterVolumeSpecName: "home") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:26.249386 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.249230    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache" (OuterVolumeSpecName: "model-cache") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:26.249386 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.249277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:26.250764 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.250726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm" (OuterVolumeSpecName: "dshm") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:26.250905 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.250872    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30bfccf4-561c-4694-ba9e-005f3488daf7" (UID: "30bfccf4-561c-4694-ba9e-005f3488daf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:26.251220 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.251202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:26.251281 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.251225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4" (OuterVolumeSpecName: "kube-api-access-257l4") pod "835aa9df-efd4-42b2-af58-6a781edbc921" (UID: "835aa9df-efd4-42b2-af58-6a781edbc921"). InnerVolumeSpecName "kube-api-access-257l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:26.251281 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.251232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg" (OuterVolumeSpecName: "kube-api-access-fvbxg") pod "30bfccf4-561c-4694-ba9e-005f3488daf7" (UID: "30bfccf4-561c-4694-ba9e-005f3488daf7"). InnerVolumeSpecName "kube-api-access-fvbxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:26.350049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350016    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350074    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvbxg\" (UniqueName: \"kubernetes.io/projected/30bfccf4-561c-4694-ba9e-005f3488daf7-kube-api-access-fvbxg\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350090    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/30bfccf4-561c-4694-ba9e-005f3488daf7-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350103    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350114    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350125    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257l4\" (UniqueName: \"kubernetes.io/projected/835aa9df-efd4-42b2-af58-6a781edbc921-kube-api-access-257l4\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350135    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835aa9df-efd4-42b2-af58-6a781edbc921-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350146    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835aa9df-efd4-42b2-af58-6a781edbc921-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.350231 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.350157    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30bfccf4-561c-4694-ba9e-005f3488daf7-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.810568 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.810535    2566 generic.go:356] "Generic (PLEG): container finished" podID="835aa9df-efd4-42b2-af58-6a781edbc921" containerID="9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f" exitCode=0
Jul 16 22:40:26.810757 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.810605    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"
Jul 16 22:40:26.810757 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.810619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" event={"ID":"835aa9df-efd4-42b2-af58-6a781edbc921","Type":"ContainerDied","Data":"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"}
Jul 16 22:40:26.810757 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.810673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm" event={"ID":"835aa9df-efd4-42b2-af58-6a781edbc921","Type":"ContainerDied","Data":"2207eb1258343da99be4347dbd4b92ab08b621dc1543bf8f154f0109f65f4463"}
Jul 16 22:40:26.810757 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.810708    2566 scope.go:117] "RemoveContainer" containerID="9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"
Jul 16 22:40:26.812054 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.812022    2566 generic.go:356] "Generic (PLEG): container finished" podID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerID="16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59" exitCode=0
Jul 16 22:40:26.812166 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.812077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"
Jul 16 22:40:26.812166 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.812118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" event={"ID":"30bfccf4-561c-4694-ba9e-005f3488daf7","Type":"ContainerDied","Data":"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"}
Jul 16 22:40:26.812275 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.812168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx" event={"ID":"30bfccf4-561c-4694-ba9e-005f3488daf7","Type":"ContainerDied","Data":"e89c2329f0ef44032343cafb553af3b839a81fa6f784517249f0b37d5d24ad91"}
Jul 16 22:40:26.819648 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.819631    2566 scope.go:117] "RemoveContainer" containerID="9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"
Jul 16 22:40:26.819942 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:40:26.819923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f\": container with ID starting with 9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f not found: ID does not exist" containerID="9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"
Jul 16 22:40:26.820019 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.819947    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f"} err="failed to get container status \"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f\": rpc error: code = NotFound desc = could not find container \"9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f\": container with ID starting with 9d86c9602fe98d945db0d097b81e8729305185be7edbda02b6971d45135c1a5f not found: ID does not exist"
Jul 16 22:40:26.820019 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.819962    2566 scope.go:117] "RemoveContainer" containerID="16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"
Jul 16 22:40:26.827673 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.827657    2566 scope.go:117] "RemoveContainer" containerID="16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"
Jul 16 22:40:26.827941 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:40:26.827923    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59\": container with ID starting with 16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59 not found: ID does not exist" containerID="16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"
Jul 16 22:40:26.827991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.827948    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59"} err="failed to get container status \"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59\": rpc error: code = NotFound desc = could not find container \"16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59\": container with ID starting with 16ce22a06ceaa0a5a1717fa30dca4813c1d05de0d7826ffd08730e4d13118e59 not found: ID does not exist"
Jul 16 22:40:26.840557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.840527    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:40:26.847325 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.847297    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4bf9pm"]
Jul 16 22:40:26.866222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.866193    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:40:26.872044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:26.872018    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schen59vx"]
Jul 16 22:40:28.319877 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:28.319843    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" path="/var/lib/kubelet/pods/30bfccf4-561c-4694-ba9e-005f3488daf7/volumes"
Jul 16 22:40:28.320269 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:28.320213    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835aa9df-efd4-42b2-af58-6a781edbc921" path="/var/lib/kubelet/pods/835aa9df-efd4-42b2-af58-6a781edbc921/volumes"
Jul 16 22:40:39.257591 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257552    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257891    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835aa9df-efd4-42b2-af58-6a781edbc921" containerName="main"
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257905    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="835aa9df-efd4-42b2-af58-6a781edbc921" containerName="main"
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257923    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerName="main"
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257929    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerName="main"
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257982    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bfccf4-561c-4694-ba9e-005f3488daf7" containerName="main"
Jul 16 22:40:39.258047 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.257993    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="835aa9df-efd4-42b2-af58-6a781edbc921" containerName="main"
Jul 16 22:40:39.260828 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.260802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.265896 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.265875    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:40:39.266293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.266270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:40:39.266451 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.266322    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 16 22:40:39.266922 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.266903    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:40:39.305986 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.305951    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:40:39.362199 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.362199 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.362488 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cml2j\" (UniqueName: \"kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.362488 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.362488 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.362488 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.362396    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463120 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463120 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cml2j\" (UniqueName: \"kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463594 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463656 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.463656 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.463622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.465717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.465696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.465900 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.465885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.474951 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.474920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cml2j\" (UniqueName: \"kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.571229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.571138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:39.719444 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.719409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:40:39.722550 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:40:39.722519    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacabb323_6b9c_4da9_afcc_26815db763ea.slice/crio-5cb22cd9a68254744e4043bf3a27c4c11d76277f40dbe8100b303ef668d9bb21 WatchSource:0}: Error finding container 5cb22cd9a68254744e4043bf3a27c4c11d76277f40dbe8100b303ef668d9bb21: Status 404 returned error can't find the container with id 5cb22cd9a68254744e4043bf3a27c4c11d76277f40dbe8100b303ef668d9bb21
Jul 16 22:40:39.780000 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.779974    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:40:39.784585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.784564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.788099 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.788081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-ddwqs\""
Jul 16 22:40:39.807462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.807426    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:40:39.866319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.866187    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.866489 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.866331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98jh\" (UniqueName: \"kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.866489 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.866375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.874610 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.874577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" event={"ID":"acabb323-6b9c-4da9-afcc-26815db763ea","Type":"ContainerStarted","Data":"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"}
Jul 16 22:40:39.874773 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.874627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" event={"ID":"acabb323-6b9c-4da9-afcc-26815db763ea","Type":"ContainerStarted","Data":"5cb22cd9a68254744e4043bf3a27c4c11d76277f40dbe8100b303ef668d9bb21"}
Jul 16 22:40:39.909049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.908996    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" podStartSLOduration=0.908979832 podStartE2EDuration="908.979832ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:39.908360141 +0000 UTC m=+3040.003871459" watchObservedRunningTime="2026-07-16 22:40:39.908979832 +0000 UTC m=+3040.004491149"
Jul 16 22:40:39.967535 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.967496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.967748 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.967572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.967748 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.967616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98jh\" (UniqueName: \"kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.968079 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.968058    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.970214 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.970194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:39.981653 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:39.981629    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98jh\" (UniqueName: \"kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:40.094004 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.093951    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:40.246189 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.246164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:40:40.248168 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:40:40.248143    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f4cad4_1f19_4dd0_ab19_30010461fadc.slice/crio-2bf77e0a634cfebe5364a430fb9577c4e804cf8f399bf715964a661f5b9c4886 WatchSource:0}: Error finding container 2bf77e0a634cfebe5364a430fb9577c4e804cf8f399bf715964a661f5b9c4886: Status 404 returned error can't find the container with id 2bf77e0a634cfebe5364a430fb9577c4e804cf8f399bf715964a661f5b9c4886
Jul 16 22:40:40.571857 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.571800    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:40.881168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.881069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" event={"ID":"51f4cad4-1f19-4dd0-ab19-30010461fadc","Type":"ContainerStarted","Data":"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"}
Jul 16 22:40:40.881168 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.881134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" event={"ID":"51f4cad4-1f19-4dd0-ab19-30010461fadc","Type":"ContainerStarted","Data":"2bf77e0a634cfebe5364a430fb9577c4e804cf8f399bf715964a661f5b9c4886"}
Jul 16 22:40:40.881404 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:40.881381    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:40:49.571666 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:49.571631    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:49.584384 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:49.584358    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:49.610428 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:49.610379    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" podStartSLOduration=10.610362966 podStartE2EDuration="10.610362966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:40.924543401 +0000 UTC m=+3041.020054719" watchObservedRunningTime="2026-07-16 22:40:49.610362966 +0000 UTC m=+3049.705874283"
Jul 16 22:40:49.937592 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:49.937513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:40:55.278202 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.278124    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:40:55.281723 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.281702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.285007 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.284986    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:40:55.286152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.286138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-epp-sa-dockercfg-bvlfm\""
Jul 16 22:40:55.286258 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.286173    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 16 22:40:55.295674 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.295647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:40:55.393810 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.393764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6kt\" (UniqueName: \"kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.394022 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.393854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.394022 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.393879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.394022 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.393976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6kt\" (UniqueName: \"kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495181 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495421 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495421 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495637 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.495765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.495640    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.497889 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.497866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.504890 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.504868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6kt\" (UniqueName: \"kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt\") pod \"tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.591955 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.591860    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:55.733462 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.733436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:40:55.735428 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:40:55.735399    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff4858f_4921_46dc_8001_205a99aae5eb.slice/crio-c84bc9024d01648b2c57102c0bc99c56bd646e5008aa9be922e01740eb2cbd37 WatchSource:0}: Error finding container c84bc9024d01648b2c57102c0bc99c56bd646e5008aa9be922e01740eb2cbd37: Status 404 returned error can't find the container with id c84bc9024d01648b2c57102c0bc99c56bd646e5008aa9be922e01740eb2cbd37
Jul 16 22:40:55.957303 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.957268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerStarted","Data":"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"}
Jul 16 22:40:55.957468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:55.957321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerStarted","Data":"c84bc9024d01648b2c57102c0bc99c56bd646e5008aa9be922e01740eb2cbd37"}
Jul 16 22:40:56.963180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:56.963145    2566 generic.go:356] "Generic (PLEG): container finished" podID="dff4858f-4921-46dc-8001-205a99aae5eb" containerID="487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8" exitCode=0
Jul 16 22:40:56.963591 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:56.963237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerDied","Data":"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"}
Jul 16 22:40:57.970557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:57.970519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerStarted","Data":"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"}
Jul 16 22:40:57.971013 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:57.970638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:40:57.997636 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:40:57.997579    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" podStartSLOduration=2.261552799 podStartE2EDuration="2.997564205s" totalImagesPullingTime="0s" totalInitContainerRuntime="736.011406ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:57.995572282 +0000 UTC m=+3058.091083610" watchObservedRunningTime="2026-07-16 22:40:57.997564205 +0000 UTC m=+3058.093075522"
Jul 16 22:41:11.887319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:11.887287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:41:13.242991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:13.242951    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:41:13.244692 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:13.244666    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:41:21.887308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:21.887260    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:41:28.264098 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.264031    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerName="main" containerID="cri-o://5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190" gracePeriod=45
Jul 16 22:41:28.266251 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.266223    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" podUID="acabb323-6b9c-4da9-afcc-26815db763ea" containerName="main" containerID="cri-o://179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89" gracePeriod=45
Jul 16 22:41:28.535064 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.535037    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:41:28.537641 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.537619    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:41:28.684503 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684503 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs\") pod \"51f4cad4-1f19-4dd0-ab19-30010461fadc\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684577    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s98jh\" (UniqueName: \"kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh\") pod \"51f4cad4-1f19-4dd0-ab19-30010461fadc\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684641    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cml2j\" (UniqueName: \"kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684694    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm\") pod \"acabb323-6b9c-4da9-afcc-26815db763ea\" (UID: \"acabb323-6b9c-4da9-afcc-26815db763ea\") "
Jul 16 22:41:28.684766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds\") pod \"51f4cad4-1f19-4dd0-ab19-30010461fadc\" (UID: \"51f4cad4-1f19-4dd0-ab19-30010461fadc\") "
Jul 16 22:41:28.685174 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.684784    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:28.685476 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.685432    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home" (OuterVolumeSpecName: "home") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:28.685476 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.685231    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:28.685692 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.685655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "51f4cad4-1f19-4dd0-ab19-30010461fadc" (UID: "51f4cad4-1f19-4dd0-ab19-30010461fadc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:28.687161 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.686132    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.689386 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.689354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51f4cad4-1f19-4dd0-ab19-30010461fadc" (UID: "51f4cad4-1f19-4dd0-ab19-30010461fadc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:41:28.694128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.694093    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm" (OuterVolumeSpecName: "dshm") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:28.694128 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.694120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:41:28.694306 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.694131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh" (OuterVolumeSpecName: "kube-api-access-s98jh") pod "51f4cad4-1f19-4dd0-ab19-30010461fadc" (UID: "51f4cad4-1f19-4dd0-ab19-30010461fadc"). InnerVolumeSpecName "kube-api-access-s98jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:28.694306 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.694140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j" (OuterVolumeSpecName: "kube-api-access-cml2j") pod "acabb323-6b9c-4da9-afcc-26815db763ea" (UID: "acabb323-6b9c-4da9-afcc-26815db763ea"). InnerVolumeSpecName "kube-api-access-cml2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786752    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acabb323-6b9c-4da9-afcc-26815db763ea-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786788    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98jh\" (UniqueName: \"kubernetes.io/projected/51f4cad4-1f19-4dd0-ab19-30010461fadc-kube-api-access-s98jh\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786799    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786808    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cml2j\" (UniqueName: \"kubernetes.io/projected/acabb323-6b9c-4da9-afcc-26815db763ea-kube-api-access-cml2j\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786833    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786846    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51f4cad4-1f19-4dd0-ab19-30010461fadc-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786858    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acabb323-6b9c-4da9-afcc-26815db763ea-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.786873 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.786868    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f4cad4-1f19-4dd0-ab19-30010461fadc-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:41:28.976401 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:28.976372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:41:29.117197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.117089    2566 generic.go:356] "Generic (PLEG): container finished" podID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerID="5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190" exitCode=0
Jul 16 22:41:29.117197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.117168    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"
Jul 16 22:41:29.117197 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.117182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" event={"ID":"51f4cad4-1f19-4dd0-ab19-30010461fadc","Type":"ContainerDied","Data":"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"}
Jul 16 22:41:29.117478 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.117239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj" event={"ID":"51f4cad4-1f19-4dd0-ab19-30010461fadc","Type":"ContainerDied","Data":"2bf77e0a634cfebe5364a430fb9577c4e804cf8f399bf715964a661f5b9c4886"}
Jul 16 22:41:29.117478 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.117266    2566 scope.go:117] "RemoveContainer" containerID="5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"
Jul 16 22:41:29.118582 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.118539    2566 generic.go:356] "Generic (PLEG): container finished" podID="acabb323-6b9c-4da9-afcc-26815db763ea" containerID="179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89" exitCode=0
Jul 16 22:41:29.118690 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.118605    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"
Jul 16 22:41:29.118690 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.118590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" event={"ID":"acabb323-6b9c-4da9-afcc-26815db763ea","Type":"ContainerDied","Data":"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"}
Jul 16 22:41:29.118767 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.118716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96" event={"ID":"acabb323-6b9c-4da9-afcc-26815db763ea","Type":"ContainerDied","Data":"5cb22cd9a68254744e4043bf3a27c4c11d76277f40dbe8100b303ef668d9bb21"}
Jul 16 22:41:29.126369 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.126351    2566 scope.go:117] "RemoveContainer" containerID="5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"
Jul 16 22:41:29.126663 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:41:29.126642    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190\": container with ID starting with 5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190 not found: ID does not exist" containerID="5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"
Jul 16 22:41:29.126750 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.126674    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190"} err="failed to get container status \"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190\": rpc error: code = NotFound desc = could not find container \"5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190\": container with ID starting with 5fe1376b2eef61e4f968c29e8510ff9982d9203153f8bfa151a895f4cc041190 not found: ID does not exist"
Jul 16 22:41:29.126750 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.126697    2566 scope.go:117] "RemoveContainer" containerID="179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"
Jul 16 22:41:29.135194 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.135166    2566 scope.go:117] "RemoveContainer" containerID="179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"
Jul 16 22:41:29.135453 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:41:29.135434    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89\": container with ID starting with 179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89 not found: ID does not exist" containerID="179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"
Jul 16 22:41:29.135500 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.135461    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89"} err="failed to get container status \"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89\": rpc error: code = NotFound desc = could not find container \"179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89\": container with ID starting with 179f8d938ff0099860f0c349381545d688f7031866912c0b2bb7e00726ba9c89 not found: ID does not exist"
Jul 16 22:41:29.150859 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.150801    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:41:29.158555 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.158523    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-6cx96"]
Jul 16 22:41:29.173278 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.173247    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:41:29.177688 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:29.177664    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-router-scheduler-669ffwt8cj"]
Jul 16 22:41:30.320797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:30.320764    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" path="/var/lib/kubelet/pods/51f4cad4-1f19-4dd0-ab19-30010461fadc/volumes"
Jul 16 22:41:30.321206 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:30.321191    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acabb323-6b9c-4da9-afcc-26815db763ea" path="/var/lib/kubelet/pods/acabb323-6b9c-4da9-afcc-26815db763ea/volumes"
Jul 16 22:41:44.122472 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.122427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:41:44.122991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.122969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerName="main"
Jul 16 22:41:44.123087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.122993    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerName="main"
Jul 16 22:41:44.123087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.123029    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acabb323-6b9c-4da9-afcc-26815db763ea" containerName="main"
Jul 16 22:41:44.123087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.123039    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="acabb323-6b9c-4da9-afcc-26815db763ea" containerName="main"
Jul 16 22:41:44.123259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.123126    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f4cad4-1f19-4dd0-ab19-30010461fadc" containerName="main"
Jul 16 22:41:44.123259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.123140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="acabb323-6b9c-4da9-afcc-26815db763ea" containerName="main"
Jul 16 22:41:44.126380 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.126359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.129721 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.129697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hzzm6\""
Jul 16 22:41:44.130966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.130948    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 16 22:41:44.138651 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.138624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:41:44.205269 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.205232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gqmp\" (UniqueName: \"kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.205439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.205273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.205439 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.205330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.306492 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.306453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gqmp\" (UniqueName: \"kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.306492 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.306494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.306730 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.306526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.307003 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.306981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.309216 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.309194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.334402 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.334366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gqmp\" (UniqueName: \"kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.436482 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.436402    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:44.590861 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:44.590830    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:41:44.594282 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:41:44.594242    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a942b6d_4e5e_4330_b76e_b7b6e4bf9d7a.slice/crio-815eddc680d3d89a9a26a55fcf133ff1b50c7454bfd7ed53e07ab0e754b8a51a WatchSource:0}: Error finding container 815eddc680d3d89a9a26a55fcf133ff1b50c7454bfd7ed53e07ab0e754b8a51a: Status 404 returned error can't find the container with id 815eddc680d3d89a9a26a55fcf133ff1b50c7454bfd7ed53e07ab0e754b8a51a
Jul 16 22:41:45.194611 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:45.194579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" event={"ID":"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a","Type":"ContainerStarted","Data":"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"}
Jul 16 22:41:45.195043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:45.194634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" event={"ID":"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a","Type":"ContainerStarted","Data":"815eddc680d3d89a9a26a55fcf133ff1b50c7454bfd7ed53e07ab0e754b8a51a"}
Jul 16 22:41:45.195043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:45.194656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:41:45.220384 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:41:45.220331    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" podStartSLOduration=1.22031551 podStartE2EDuration="1.22031551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:45.218955409 +0000 UTC m=+3105.314466726" watchObservedRunningTime="2026-07-16 22:41:45.22031551 +0000 UTC m=+3105.315826827"
Jul 16 22:42:16.201304 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:16.201275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:42:18.461656 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:18.461622    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:42:26.202243 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:26.202156    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:42:28.941471 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:28.941438    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 16 22:42:28.941754 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:42:28.941472    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main" containerID="cri-o://12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"
Jul 16 22:42:28.941754 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:28.941514    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main" containerID="cri-o://12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c" gracePeriod=50
Jul 16 22:42:29.086717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.086696    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k_1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a/main/0.log"
Jul 16 22:42:29.089094 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.089077    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:42:29.194078 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.193982    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs\") pod \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") "
Jul 16 22:42:29.194260 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.194116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gqmp\" (UniqueName: \"kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp\") pod \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") "
Jul 16 22:42:29.194260 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.194142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds\") pod \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\" (UID: \"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a\") "
Jul 16 22:42:29.194486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.194456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" (UID: "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:29.196257 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.196237    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp" (OuterVolumeSpecName: "kube-api-access-9gqmp") pod "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" (UID: "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a"). InnerVolumeSpecName "kube-api-access-9gqmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:42:29.196336 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.196261    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" (UID: "1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:42:29.295044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.295007    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gqmp\" (UniqueName: \"kubernetes.io/projected/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-kube-api-access-9gqmp\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:42:29.295044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.295039    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:42:29.295044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.295048    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:42:29.397363 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397336    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k_1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a/main/0.log"
Jul 16 22:42:29.397562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397374    2566 generic.go:356] "Generic (PLEG): container finished" podID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerID="12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c" exitCode=1
Jul 16 22:42:29.397562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" event={"ID":"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a","Type":"ContainerDied","Data":"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"}
Jul 16 22:42:29.397562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397445    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"
Jul 16 22:42:29.397562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397464    2566 scope.go:117] "RemoveContainer" containerID="12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"
Jul 16 22:42:29.397562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.397454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k" event={"ID":"1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a","Type":"ContainerDied","Data":"815eddc680d3d89a9a26a55fcf133ff1b50c7454bfd7ed53e07ab0e754b8a51a"}
Jul 16 22:42:29.406523 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.406506    2566 scope.go:117] "RemoveContainer" containerID="12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"
Jul 16 22:42:29.406773 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:42:29.406755    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c\": container with ID starting with 12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c not found: ID does not exist" containerID="12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"
Jul 16 22:42:29.406838 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.406781    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c"} err="failed to get container status \"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c\": rpc error: code = NotFound desc = could not find container \"12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c\": container with ID starting with 12aeecf8a8933c97a65b16a77570ce45adc24681f256dda508253cb970b3f67c not found: ID does not exist"
Jul 16 22:42:29.440092 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.440058    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:42:29.450040 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:29.449968    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fbhn5k"]
Jul 16 22:42:30.319861 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:30.319802    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" path="/var/lib/kubelet/pods/1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a/volumes"
Jul 16 22:42:45.807776 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.807747    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:42:45.810180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.808117    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main"
Jul 16 22:42:45.810180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.808129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main"
Jul 16 22:42:45.810180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.808195    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a942b6d-4e5e-4330-b76e-b7b6e4bf9d7a" containerName="main"
Jul 16 22:42:45.811091 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.811065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.814321 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.814302    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 16 22:42:45.829531 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.829650 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.829717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829663    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.829717 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:42:45.829800 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87v9c\" (UniqueName: \"kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.829800 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.829920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.829854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.930991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.930961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.930991 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87v9c\" (UniqueName: \"kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931247 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931247 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931247 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931404 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931478 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931439    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.931645 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.931624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.933473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.933441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.933836 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.933798    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:45.947900 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:45.947875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87v9c\" (UniqueName: \"kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:46.121564 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:46.121471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:46.271637 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:46.271613    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:42:46.274351 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:42:46.274326    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a880b87_cddd_4ff6_891a_9d6ac8b52727.slice/crio-c6e6276e256b29264cb2102673afe3d4092891a634ce8adae1578866b073fe91 WatchSource:0}: Error finding container c6e6276e256b29264cb2102673afe3d4092891a634ce8adae1578866b073fe91: Status 404 returned error can't find the container with id c6e6276e256b29264cb2102673afe3d4092891a634ce8adae1578866b073fe91
Jul 16 22:42:46.475698 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:46.475661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" event={"ID":"3a880b87-cddd-4ff6-891a-9d6ac8b52727","Type":"ContainerStarted","Data":"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"}
Jul 16 22:42:46.475897 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:46.475717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" event={"ID":"3a880b87-cddd-4ff6-891a-9d6ac8b52727","Type":"ContainerStarted","Data":"c6e6276e256b29264cb2102673afe3d4092891a634ce8adae1578866b073fe91"}
Jul 16 22:42:46.509562 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:46.509510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" podStartSLOduration=1.509495196 podStartE2EDuration="1.509495196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:46.508266292 +0000 UTC m=+3166.603777609" watchObservedRunningTime="2026-07-16 22:42:46.509495196 +0000 UTC m=+3166.605006580"
Jul 16 22:42:47.122598 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:47.122558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:56.122374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:56.122330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:56.135601 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:56.135570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:56.542266 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:56.542230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:42:59.638639 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:42:59.638604    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:43:08.977269 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:08.977227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:43:14.659473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.659427    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="main" containerID="cri-o://f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be" gracePeriod=45
Jul 16 22:43:14.909749 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.909692    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:43:14.977834 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.977781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location\") pod \"dff4858f-4921-46dc-8001-205a99aae5eb\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") "
Jul 16 22:43:14.978017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.977900    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs\") pod \"dff4858f-4921-46dc-8001-205a99aae5eb\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") "
Jul 16 22:43:14.978017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.977932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds\") pod \"dff4858f-4921-46dc-8001-205a99aae5eb\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") "
Jul 16 22:43:14.978017 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.977972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6kt\" (UniqueName: \"kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt\") pod \"dff4858f-4921-46dc-8001-205a99aae5eb\" (UID: \"dff4858f-4921-46dc-8001-205a99aae5eb\") "
Jul 16 22:43:14.978259 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.978223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "dff4858f-4921-46dc-8001-205a99aae5eb" (UID: "dff4858f-4921-46dc-8001-205a99aae5eb"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:14.978590 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.978567    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dff4858f-4921-46dc-8001-205a99aae5eb" (UID: "dff4858f-4921-46dc-8001-205a99aae5eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:14.980258 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.980233    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dff4858f-4921-46dc-8001-205a99aae5eb" (UID: "dff4858f-4921-46dc-8001-205a99aae5eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:43:14.980258 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:14.980239    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt" (OuterVolumeSpecName: "kube-api-access-6q6kt") pod "dff4858f-4921-46dc-8001-205a99aae5eb" (UID: "dff4858f-4921-46dc-8001-205a99aae5eb"). InnerVolumeSpecName "kube-api-access-6q6kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:15.079195 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.079125    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff4858f-4921-46dc-8001-205a99aae5eb-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.079195 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.079188    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.079195 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.079200    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6kt\" (UniqueName: \"kubernetes.io/projected/dff4858f-4921-46dc-8001-205a99aae5eb-kube-api-access-6q6kt\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.079422 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.079210    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dff4858f-4921-46dc-8001-205a99aae5eb-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:15.621599 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.621569    2566 generic.go:356] "Generic (PLEG): container finished" podID="dff4858f-4921-46dc-8001-205a99aae5eb" containerID="f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be" exitCode=0
Jul 16 22:43:15.621766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.621618    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"
Jul 16 22:43:15.621766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.621631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerDied","Data":"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"}
Jul 16 22:43:15.621766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.621687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v" event={"ID":"dff4858f-4921-46dc-8001-205a99aae5eb","Type":"ContainerDied","Data":"c84bc9024d01648b2c57102c0bc99c56bd646e5008aa9be922e01740eb2cbd37"}
Jul 16 22:43:15.621766 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.621715    2566 scope.go:117] "RemoveContainer" containerID="f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"
Jul 16 22:43:15.630909 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.630889    2566 scope.go:117] "RemoveContainer" containerID="487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"
Jul 16 22:43:15.638445 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.638429    2566 scope.go:117] "RemoveContainer" containerID="f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"
Jul 16 22:43:15.638704 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:43:15.638684    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be\": container with ID starting with f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be not found: ID does not exist" containerID="f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"
Jul 16 22:43:15.638742 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.638712    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be"} err="failed to get container status \"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be\": rpc error: code = NotFound desc = could not find container \"f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be\": container with ID starting with f705d0ca63d063c984eaeb7e52d66f6724b7823b2813f5e74606540da2d510be not found: ID does not exist"
Jul 16 22:43:15.638742 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.638730    2566 scope.go:117] "RemoveContainer" containerID="487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"
Jul 16 22:43:15.638997 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:43:15.638980    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8\": container with ID starting with 487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8 not found: ID does not exist" containerID="487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"
Jul 16 22:43:15.639039 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.639004    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8"} err="failed to get container status \"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8\": rpc error: code = NotFound desc = could not find container \"487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8\": container with ID starting with 487c03e1f2c173bf5ec4b50b785a12f5ca6e21e1be3b5506875dd7648d1bb4c8 not found: ID does not exist"
Jul 16 22:43:15.651835 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.647992    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:43:15.654247 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:15.654221    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-router-scheduler-5f8d594d4b-q8h9v"]
Jul 16 22:43:16.319833 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:16.319787    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" path="/var/lib/kubelet/pods/dff4858f-4921-46dc-8001-205a99aae5eb/volumes"
Jul 16 22:43:24.716674 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.716639    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:43:24.717083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.717007    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="main"
Jul 16 22:43:24.717083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.717019    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="main"
Jul 16 22:43:24.717083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.717032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="storage-initializer"
Jul 16 22:43:24.717083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.717038    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="storage-initializer"
Jul 16 22:43:24.717083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.717085    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff4858f-4921-46dc-8001-205a99aae5eb" containerName="main"
Jul 16 22:43:24.721680 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.721660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.724851 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.724808    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 16 22:43:24.725758 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.725734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-vts9f\""
Jul 16 22:43:24.725758 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.725754    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 22:43:24.732323 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.732301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:43:24.860172 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.860124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.860172 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.860176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vfh\" (UniqueName: \"kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.860433 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.860284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.860433 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.860351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.960964 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.960926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vfh\" (UniqueName: \"kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.961144 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.960989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.961144 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.961045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.961144 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.961113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.961492 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.961474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.961531 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.961465    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.963724 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.963698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:24.970033 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:24.969968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vfh\" (UniqueName: \"kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh\") pod \"prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:25.032486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:25.032446    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:25.213954 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:25.213928    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:43:25.215929 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:43:25.215898    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6f6cc3_37ee_4081_bcc6_5feda1fef33a.slice/crio-f9bbb713364098ef9fd72a2e91ef07de34f06735c33c0d494d06a61bb4ca1c70 WatchSource:0}: Error finding container f9bbb713364098ef9fd72a2e91ef07de34f06735c33c0d494d06a61bb4ca1c70: Status 404 returned error can't find the container with id f9bbb713364098ef9fd72a2e91ef07de34f06735c33c0d494d06a61bb4ca1c70
Jul 16 22:43:25.673244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:25.673146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerStarted","Data":"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"}
Jul 16 22:43:25.673244 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:25.673213    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerStarted","Data":"f9bbb713364098ef9fd72a2e91ef07de34f06735c33c0d494d06a61bb4ca1c70"}
Jul 16 22:43:26.679742 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:26.679703    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerID="754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88" exitCode=0
Jul 16 22:43:26.680192 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:26.679783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerDied","Data":"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"}
Jul 16 22:43:27.686961 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:27.686916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerStarted","Data":"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"}
Jul 16 22:43:27.687427 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:27.687011    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:43:27.711299 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:27.711248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" podStartSLOduration=2.9317929 podStartE2EDuration="3.711233409s" totalImagesPullingTime="0s" totalInitContainerRuntime="779.440509ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:27.710032 +0000 UTC m=+3207.805543319" watchObservedRunningTime="2026-07-16 22:43:27.711233409 +0000 UTC m=+3207.806744726"
Jul 16 22:43:41.742290 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:41.742254    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:43:56.765612 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:56.765513    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" podUID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" containerName="main" containerID="cri-o://e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102" gracePeriod=45
Jul 16 22:43:57.024845 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.024762    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:43:57.131055 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131055 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131064    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131087    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87v9c\" (UniqueName: \"kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131204    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache\") pod \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\" (UID: \"3a880b87-cddd-4ff6-891a-9d6ac8b52727\") "
Jul 16 22:43:57.131520 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131418    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home" (OuterVolumeSpecName: "home") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:57.131577 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:57.131612 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.131579    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:57.133434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.133407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c" (OuterVolumeSpecName: "kube-api-access-87v9c") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "kube-api-access-87v9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:43:57.133783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.133766    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:43:57.133872 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.133769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm" (OuterVolumeSpecName: "dshm") pod "3a880b87-cddd-4ff6-891a-9d6ac8b52727" (UID: "3a880b87-cddd-4ff6-891a-9d6ac8b52727"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:43:57.232621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232576    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87v9c\" (UniqueName: \"kubernetes.io/projected/3a880b87-cddd-4ff6-891a-9d6ac8b52727-kube-api-access-87v9c\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.232621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232613    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.232621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232624    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.232621 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232632    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.232938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232640    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a880b87-cddd-4ff6-891a-9d6ac8b52727-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.232938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.232648    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a880b87-cddd-4ff6-891a-9d6ac8b52727-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:43:57.826584 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.826549    2566 generic.go:356] "Generic (PLEG): container finished" podID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" containerID="e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102" exitCode=0
Jul 16 22:43:57.827004 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.826597    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"
Jul 16 22:43:57.827004 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.826628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" event={"ID":"3a880b87-cddd-4ff6-891a-9d6ac8b52727","Type":"ContainerDied","Data":"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"}
Jul 16 22:43:57.827004 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.826688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8" event={"ID":"3a880b87-cddd-4ff6-891a-9d6ac8b52727","Type":"ContainerDied","Data":"c6e6276e256b29264cb2102673afe3d4092891a634ce8adae1578866b073fe91"}
Jul 16 22:43:57.827004 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.826719    2566 scope.go:117] "RemoveContainer" containerID="e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"
Jul 16 22:43:57.836175 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.836159    2566 scope.go:117] "RemoveContainer" containerID="e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"
Jul 16 22:43:57.836449 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:43:57.836427    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102\": container with ID starting with e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102 not found: ID does not exist" containerID="e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"
Jul 16 22:43:57.836516 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.836463    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102"} err="failed to get container status \"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102\": rpc error: code = NotFound desc = could not find container \"e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102\": container with ID starting with e9d3c3c735a134f9cfbb50cca661343a7b873bc789b8b6dd9c24c83b10c05102 not found: ID does not exist"
Jul 16 22:43:57.855425 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.855393    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:43:57.866344 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:57.866313    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-f8wm8"]
Jul 16 22:43:58.319927 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:58.319894    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" path="/var/lib/kubelet/pods/3a880b87-cddd-4ff6-891a-9d6ac8b52727/volumes"
Jul 16 22:43:58.694557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:43:58.694473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:44:07.587465 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.587427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:07.587895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.587764    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" containerName="main"
Jul 16 22:44:07.587895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.587775    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" containerName="main"
Jul 16 22:44:07.587895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.587872    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a880b87-cddd-4ff6-891a-9d6ac8b52727" containerName="main"
Jul 16 22:44:07.592604 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.592574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.595918 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.595889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 16 22:44:07.609719 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.609689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:07.724545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.724545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.724753 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.724753 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtn4z\" (UniqueName: \"kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.724753 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.724753 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.724704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826264 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826264 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtn4z\" (UniqueName: \"kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826349 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826349 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826482 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826544 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.826544 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.826524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.828805 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.828782    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.829222 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.829197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.846160 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.846082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtn4z\" (UniqueName: \"kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.886293 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.886260    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:07.889893 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.889876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:07.893459 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.893437    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-zxd9t\""
Jul 16 22:44:07.902469 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.902434    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:07.902591 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.902555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:07.927738 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.927692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:07.927938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.927749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:07.927938 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:07.927836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvcwl\" (UniqueName: \"kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.028704 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.028673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.028704 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.028718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.029012 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.028766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvcwl\" (UniqueName: \"kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.029151 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.029129    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.031430 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.031372    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.058270 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.058185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:08.060733 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:44:08.060707    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2ac7e0_edc4_4297_8062_03d60739ded2.slice/crio-b032d132b7994b499d21c4432268cc82fe409e6e60d661b0fadd4d099afb86e9 WatchSource:0}: Error finding container b032d132b7994b499d21c4432268cc82fe409e6e60d661b0fadd4d099afb86e9: Status 404 returned error can't find the container with id b032d132b7994b499d21c4432268cc82fe409e6e60d661b0fadd4d099afb86e9
Jul 16 22:44:08.060965 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.060945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvcwl\" (UniqueName: \"kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx\" (UID: \"73970152-802c-4611-928f-646c8487decf\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.199922 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.199894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.343720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.343674    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:08.349261 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:44:08.349233    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73970152_802c_4611_928f_646c8487decf.slice/crio-aad71aecbde925fb6ab0c322e337a9e814a60b9e71edf30614bcf9bbe40f8d39 WatchSource:0}: Error finding container aad71aecbde925fb6ab0c322e337a9e814a60b9e71edf30614bcf9bbe40f8d39: Status 404 returned error can't find the container with id aad71aecbde925fb6ab0c322e337a9e814a60b9e71edf30614bcf9bbe40f8d39
Jul 16 22:44:08.881895 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.881858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" event={"ID":"73970152-802c-4611-928f-646c8487decf","Type":"ContainerStarted","Data":"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"}
Jul 16 22:44:08.882330 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.881909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" event={"ID":"73970152-802c-4611-928f-646c8487decf","Type":"ContainerStarted","Data":"aad71aecbde925fb6ab0c322e337a9e814a60b9e71edf30614bcf9bbe40f8d39"}
Jul 16 22:44:08.882330 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.882020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:08.883346 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.883322    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" event={"ID":"4d2ac7e0-edc4-4297-8062-03d60739ded2","Type":"ContainerStarted","Data":"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"}
Jul 16 22:44:08.883452 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.883358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" event={"ID":"4d2ac7e0-edc4-4297-8062-03d60739ded2","Type":"ContainerStarted","Data":"b032d132b7994b499d21c4432268cc82fe409e6e60d661b0fadd4d099afb86e9"}
Jul 16 22:44:08.903261 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.903212    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" podStartSLOduration=1.903193895 podStartE2EDuration="1.903193895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:08.902109449 +0000 UTC m=+3248.997620767" watchObservedRunningTime="2026-07-16 22:44:08.903193895 +0000 UTC m=+3248.998705216"
Jul 16 22:44:08.903373 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.903268    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:08.923166 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:08.923110    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" podStartSLOduration=1.923091271 podStartE2EDuration="1.923091271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:08.920910026 +0000 UTC m=+3249.016421340" watchObservedRunningTime="2026-07-16 22:44:08.923091271 +0000 UTC m=+3249.018602591"
Jul 16 22:44:17.903511 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:17.903474    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:17.916058 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:17.916022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:17.936980 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:17.936956    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:39.891273 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:39.891242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:41.686026 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:41.685989    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:41.693381 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:41.693352    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:49.891224 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:49.891184    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" podUID="73970152-802c-4611-928f-646c8487decf" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:44:56.708318 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:56.708248    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerName="main" containerID="cri-o://68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e" gracePeriod=45
Jul 16 22:44:56.715030 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:56.714980    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" podUID="73970152-802c-4611-928f-646c8487decf" containerName="main" containerID="cri-o://9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f" gracePeriod=45
Jul 16 22:44:56.982788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:56.982758    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:56.986262 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:56.986240    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:57.048316 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvcwl\" (UniqueName: \"kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl\") pod \"73970152-802c-4611-928f-646c8487decf\" (UID: \"73970152-802c-4611-928f-646c8487decf\") "
Jul 16 22:44:57.048486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048325    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtn4z\" (UniqueName: \"kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.048486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs\") pod \"73970152-802c-4611-928f-646c8487decf\" (UID: \"73970152-802c-4611-928f-646c8487decf\") "
Jul 16 22:44:57.048486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048412    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.048486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.048486 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.048667 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds\") pod \"73970152-802c-4611-928f-646c8487decf\" (UID: \"73970152-802c-4611-928f-646c8487decf\") "
Jul 16 22:44:57.048736 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048702    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:57.048848 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "73970152-802c-4611-928f-646c8487decf" (UID: "73970152-802c-4611-928f-646c8487decf"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:57.048914 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.048962 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.048947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache\") pod \"4d2ac7e0-edc4-4297-8062-03d60739ded2\" (UID: \"4d2ac7e0-edc4-4297-8062-03d60739ded2\") "
Jul 16 22:44:57.049028 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home" (OuterVolumeSpecName: "home") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:57.049512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:57.049512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049428    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/73970152-802c-4611-928f-646c8487decf-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.049512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049447    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.049512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049460    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.049512 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.049473    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.050926 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.050899    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z" (OuterVolumeSpecName: "kube-api-access-qtn4z") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "kube-api-access-qtn4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:57.051052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.050947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm" (OuterVolumeSpecName: "dshm") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:57.051052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.051029    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl" (OuterVolumeSpecName: "kube-api-access-mvcwl") pod "73970152-802c-4611-928f-646c8487decf" (UID: "73970152-802c-4611-928f-646c8487decf"). InnerVolumeSpecName "kube-api-access-mvcwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:57.051298 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.051282    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73970152-802c-4611-928f-646c8487decf" (UID: "73970152-802c-4611-928f-646c8487decf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:57.051364 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.051298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d2ac7e0-edc4-4297-8062-03d60739ded2" (UID: "4d2ac7e0-edc4-4297-8062-03d60739ded2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:57.106107 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.106072    2566 generic.go:356] "Generic (PLEG): container finished" podID="73970152-802c-4611-928f-646c8487decf" containerID="9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f" exitCode=0
Jul 16 22:44:57.106289 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.106119    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"
Jul 16 22:44:57.106289 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.106166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" event={"ID":"73970152-802c-4611-928f-646c8487decf","Type":"ContainerDied","Data":"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"}
Jul 16 22:44:57.106289 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.106224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx" event={"ID":"73970152-802c-4611-928f-646c8487decf","Type":"ContainerDied","Data":"aad71aecbde925fb6ab0c322e337a9e814a60b9e71edf30614bcf9bbe40f8d39"}
Jul 16 22:44:57.106289 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.106253    2566 scope.go:117] "RemoveContainer" containerID="9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"
Jul 16 22:44:57.107479 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.107459    2566 generic.go:356] "Generic (PLEG): container finished" podID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerID="68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e" exitCode=0
Jul 16 22:44:57.107557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.107503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" event={"ID":"4d2ac7e0-edc4-4297-8062-03d60739ded2","Type":"ContainerDied","Data":"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"}
Jul 16 22:44:57.107557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.107510    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"
Jul 16 22:44:57.107557 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.107540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" event={"ID":"4d2ac7e0-edc4-4297-8062-03d60739ded2","Type":"ContainerDied","Data":"b032d132b7994b499d21c4432268cc82fe409e6e60d661b0fadd4d099afb86e9"}
Jul 16 22:44:57.115321 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.115301    2566 scope.go:117] "RemoveContainer" containerID="9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"
Jul 16 22:44:57.115668 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:44:57.115641    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f\": container with ID starting with 9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f not found: ID does not exist" containerID="9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"
Jul 16 22:44:57.115784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.115680    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f"} err="failed to get container status \"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f\": rpc error: code = NotFound desc = could not find container \"9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f\": container with ID starting with 9c19a0f23e6280df8aff574cba4032ec28d5018b45d8ad2213ad90ab5bcaec8f not found: ID does not exist"
Jul 16 22:44:57.115784 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.115708    2566 scope.go:117] "RemoveContainer" containerID="68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"
Jul 16 22:44:57.124898 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.124881    2566 scope.go:117] "RemoveContainer" containerID="68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"
Jul 16 22:44:57.125170 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:44:57.125151    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e\": container with ID starting with 68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e not found: ID does not exist" containerID="68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"
Jul 16 22:44:57.125226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.125176    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e"} err="failed to get container status \"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e\": rpc error: code = NotFound desc = could not find container \"68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e\": container with ID starting with 68d2d49acf40988a32006682f0d774a992155eed76648869a1a05f2ee3b4475e not found: ID does not exist"
Jul 16 22:44:57.135789 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.135761    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:57.141136 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.141112    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-6f6gggx"]
Jul 16 22:44:57.150904 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.150880    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvcwl\" (UniqueName: \"kubernetes.io/projected/73970152-802c-4611-928f-646c8487decf-kube-api-access-mvcwl\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.150904 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.150905    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtn4z\" (UniqueName: \"kubernetes.io/projected/4d2ac7e0-edc4-4297-8062-03d60739ded2-kube-api-access-qtn4z\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.151052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.150917    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73970152-802c-4611-928f-646c8487decf-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.151052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.150930    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d2ac7e0-edc4-4297-8062-03d60739ded2-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.151052 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.150938    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d2ac7e0-edc4-4297-8062-03d60739ded2-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:44:57.167085 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.167046    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:57.176602 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.176563    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp"]
Jul 16 22:44:57.927499 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:57.927448    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-x2drp" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:44:58.319972 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:58.319935    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" path="/var/lib/kubelet/pods/4d2ac7e0-edc4-4297-8062-03d60739ded2/volumes"
Jul 16 22:44:58.320339 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:44:58.320326    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73970152-802c-4611-928f-646c8487decf" path="/var/lib/kubelet/pods/73970152-802c-4611-928f-646c8487decf/volumes"
Jul 16 22:45:00.533319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:00.533297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:45:00.542284 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:00.542262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jndh2_ffa0b617-4199-4471-b583-fd0d97634b58/ovn-acl-logging/0.log"
Jul 16 22:45:06.352079 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352042    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:06.352487 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352384    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerName="main"
Jul 16 22:45:06.352487 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352394    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerName="main"
Jul 16 22:45:06.352487 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352406    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73970152-802c-4611-928f-646c8487decf" containerName="main"
Jul 16 22:45:06.352487 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352411    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="73970152-802c-4611-928f-646c8487decf" containerName="main"
Jul 16 22:45:06.352641 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352497    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="73970152-802c-4611-928f-646c8487decf" containerName="main"
Jul 16 22:45:06.352641 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.352508    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2ac7e0-edc4-4297-8062-03d60739ded2" containerName="main"
Jul 16 22:45:06.357604 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.357578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.361019 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.360992    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 16 22:45:06.372681 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.372651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:06.435077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.435077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.435327 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.435327 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtxp\" (UniqueName: \"kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.435327 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.435327 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.435260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535718 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtxp\" (UniqueName: \"kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.535920 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.535879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.536246 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.536223    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.536308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.536239    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.536308 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.536270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.538152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.538125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.538552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.538532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.545468 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.545434    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtxp\" (UniqueName: \"kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.587919 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.587877    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:06.591391 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.591368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.594279 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.594254    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-hsl9x\""
Jul 16 22:45:06.602229 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.602162    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:06.636693 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.636654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccph\" (UniqueName: \"kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.636900 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.636723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.636900 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.636747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.667891 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.667853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:06.737420 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.737387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccph\" (UniqueName: \"kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.737566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.737482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.737566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.737520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.737963 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.737945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.741553 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.741528    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.747335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.747311    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccph\" (UniqueName: \"kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:06.813219 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.813194    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:06.815939 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:45:06.815915    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c809ae9_8950_4528_b1bc_bdd0dc790cd1.slice/crio-611b4ed50c448620854771a3c2437a1cb603302672451e37aba166eec6ce99aa WatchSource:0}: Error finding container 611b4ed50c448620854771a3c2437a1cb603302672451e37aba166eec6ce99aa: Status 404 returned error can't find the container with id 611b4ed50c448620854771a3c2437a1cb603302672451e37aba166eec6ce99aa
Jul 16 22:45:06.902567 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:06.902532    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:07.073133 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.073104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:07.075497 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:45:07.075466    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5f6e87_0e0c_4be5_9be2_a91adc8c512a.slice/crio-091694ae2d8ce55d4b23bf0fe785585de254f9bf7556fced270d281981fc90e4 WatchSource:0}: Error finding container 091694ae2d8ce55d4b23bf0fe785585de254f9bf7556fced270d281981fc90e4: Status 404 returned error can't find the container with id 091694ae2d8ce55d4b23bf0fe785585de254f9bf7556fced270d281981fc90e4
Jul 16 22:45:07.156890 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.156858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" event={"ID":"1c809ae9-8950-4528-b1bc-bdd0dc790cd1","Type":"ContainerStarted","Data":"a9078d5948b0af222727adcdb01c113d7d9e1d960dcbf7eb255131b158a86069"}
Jul 16 22:45:07.157024 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.156911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" event={"ID":"1c809ae9-8950-4528-b1bc-bdd0dc790cd1","Type":"ContainerStarted","Data":"611b4ed50c448620854771a3c2437a1cb603302672451e37aba166eec6ce99aa"}
Jul 16 22:45:07.158507 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.158479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" event={"ID":"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a","Type":"ContainerStarted","Data":"508be79ca53ec2ec271b3146e81ccaacb11afda3ead959a337fe1ecc5f8a8207"}
Jul 16 22:45:07.158611 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.158528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" event={"ID":"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a","Type":"ContainerStarted","Data":"091694ae2d8ce55d4b23bf0fe785585de254f9bf7556fced270d281981fc90e4"}
Jul 16 22:45:07.158611 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.158569    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:07.189011 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.188960    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" podStartSLOduration=1.188942296 podStartE2EDuration="1.188942296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:07.187658663 +0000 UTC m=+3307.283169981" watchObservedRunningTime="2026-07-16 22:45:07.188942296 +0000 UTC m=+3307.284453613"
Jul 16 22:45:07.223388 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.223333    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" podStartSLOduration=1.2233149540000001 podStartE2EDuration="1.223314954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:07.220685675 +0000 UTC m=+3307.316196989" watchObservedRunningTime="2026-07-16 22:45:07.223314954 +0000 UTC m=+3307.318826300"
Jul 16 22:45:07.668340 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:07.668302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:16.668011 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:16.667972    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:16.680524 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:16.680500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:17.219588 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:17.219556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:38.165463 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:38.165377    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:40.223574 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:40.223538    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:40.227477 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:40.227447    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:44.439504 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:44.439467    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:45:48.165624 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:48.165584    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:45:48.693844 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:48.693779    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:45:55.245262 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.245190    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" podUID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" containerName="main" containerID="cri-o://a9078d5948b0af222727adcdb01c113d7d9e1d960dcbf7eb255131b158a86069" gracePeriod=45
Jul 16 22:45:55.247506 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.247457    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerName="main" containerID="cri-o://508be79ca53ec2ec271b3146e81ccaacb11afda3ead959a337fe1ecc5f8a8207" gracePeriod=45
Jul 16 22:45:55.388433 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.388394    2566 generic.go:356] "Generic (PLEG): container finished" podID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" containerID="a9078d5948b0af222727adcdb01c113d7d9e1d960dcbf7eb255131b158a86069" exitCode=0
Jul 16 22:45:55.388592 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.388470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" event={"ID":"1c809ae9-8950-4528-b1bc-bdd0dc790cd1","Type":"ContainerDied","Data":"a9078d5948b0af222727adcdb01c113d7d9e1d960dcbf7eb255131b158a86069"}
Jul 16 22:45:55.390291 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.390262    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerID="508be79ca53ec2ec271b3146e81ccaacb11afda3ead959a337fe1ecc5f8a8207" exitCode=0
Jul 16 22:45:55.390411 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.390307    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" event={"ID":"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a","Type":"ContainerDied","Data":"508be79ca53ec2ec271b3146e81ccaacb11afda3ead959a337fe1ecc5f8a8207"}
Jul 16 22:45:55.522427 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.522394    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:55.525394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.525373    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:55.544780 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.544741    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.544925 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.544827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.544925 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.544871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ccph\" (UniqueName: \"kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph\") pod \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") "
Jul 16 22:45:55.544925 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.544913    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds\") pod \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") "
Jul 16 22:45:55.545079 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.544952    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.545079 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs\") pod \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\" (UID: \"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a\") "
Jul 16 22:45:55.545079 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:55.545243 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545064    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtxp\" (UniqueName: \"kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.545243 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.545243 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs\") pod \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\" (UID: \"1c809ae9-8950-4528-b1bc-bdd0dc790cd1\") "
Jul 16 22:45:55.545395 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" (UID: "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:55.545395 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:55.545495 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545479    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.545553 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545500    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.545553 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.545514    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.546312 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.546259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home" (OuterVolumeSpecName: "home") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:55.547953 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.547920    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph" (OuterVolumeSpecName: "kube-api-access-2ccph") pod "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" (UID: "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a"). InnerVolumeSpecName "kube-api-access-2ccph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:45:55.548070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.547957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp" (OuterVolumeSpecName: "kube-api-access-rgtxp") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "kube-api-access-rgtxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:45:55.548070 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.547995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm" (OuterVolumeSpecName: "dshm") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:55.548196 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.548149    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" (UID: "9b5f6e87-0e0c-4be5-9be2-a91adc8c512a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:45:55.548664 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.548645    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c809ae9-8950-4528-b1bc-bdd0dc790cd1" (UID: "1c809ae9-8950-4528-b1bc-bdd0dc790cd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:45:55.646457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646398    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.646457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646449    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtxp\" (UniqueName: \"kubernetes.io/projected/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-kube-api-access-rgtxp\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.646457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646460    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.646457 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646469    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.646747 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646477    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c809ae9-8950-4528-b1bc-bdd0dc790cd1-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:55.646747 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:55.646485    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccph\" (UniqueName: \"kubernetes.io/projected/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a-kube-api-access-2ccph\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:56.396384 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.396349    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"
Jul 16 22:45:56.396384 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.396378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2" event={"ID":"1c809ae9-8950-4528-b1bc-bdd0dc790cd1","Type":"ContainerDied","Data":"611b4ed50c448620854771a3c2437a1cb603302672451e37aba166eec6ce99aa"}
Jul 16 22:45:56.396951 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.396438    2566 scope.go:117] "RemoveContainer" containerID="a9078d5948b0af222727adcdb01c113d7d9e1d960dcbf7eb255131b158a86069"
Jul 16 22:45:56.397901 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.397741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx" event={"ID":"9b5f6e87-0e0c-4be5-9be2-a91adc8c512a","Type":"ContainerDied","Data":"091694ae2d8ce55d4b23bf0fe785585de254f9bf7556fced270d281981fc90e4"}
Jul 16 22:45:56.397901 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.397760    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"
Jul 16 22:45:56.405263 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.405244    2566 scope.go:117] "RemoveContainer" containerID="508be79ca53ec2ec271b3146e81ccaacb11afda3ead959a337fe1ecc5f8a8207"
Jul 16 22:45:56.417164 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.417139    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:56.422804 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.422782    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5dmkx2"]
Jul 16 22:45:56.436700 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.436677    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:56.446828 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:56.446790    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedzx9nx"]
Jul 16 22:45:58.319614 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:58.319584    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" path="/var/lib/kubelet/pods/1c809ae9-8950-4528-b1bc-bdd0dc790cd1/volumes"
Jul 16 22:45:58.320010 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:58.319995    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" path="/var/lib/kubelet/pods/9b5f6e87-0e0c-4be5-9be2-a91adc8c512a/volumes"
Jul 16 22:45:58.694174 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:58.694091    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:45:59.459918 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.459872    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main" containerID="cri-o://fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b" gracePeriod=45
Jul 16 22:45:59.719257 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.719188    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:45:59.780776 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.780744    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs\") pod \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") "
Jul 16 22:45:59.780981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.780801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9vfh\" (UniqueName: \"kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh\") pod \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") "
Jul 16 22:45:59.780981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.780861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds\") pod \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") "
Jul 16 22:45:59.780981 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.780927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location\") pod \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\" (UID: \"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a\") "
Jul 16 22:45:59.781201 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.781157    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" (UID: "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:59.781636 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.781611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" (UID: "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:59.783109 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.783091    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh" (OuterVolumeSpecName: "kube-api-access-m9vfh") pod "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" (UID: "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a"). InnerVolumeSpecName "kube-api-access-m9vfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:45:59.783191 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.783163    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" (UID: "6f6f6cc3-37ee-4081-bcc6-5feda1fef33a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:45:59.882009 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.881971    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:59.882009 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.882003    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vfh\" (UniqueName: \"kubernetes.io/projected/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kube-api-access-m9vfh\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:59.882009 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.882014    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:45:59.882251 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:45:59.882023    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a-kserve-provision-location\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:46:00.420307 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.420269    2566 generic.go:356] "Generic (PLEG): container finished" podID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerID="fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b" exitCode=0
Jul 16 22:46:00.420307 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.420311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerDied","Data":"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"}
Jul 16 22:46:00.420539 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.420331    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"
Jul 16 22:46:00.420539 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.420353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k" event={"ID":"6f6f6cc3-37ee-4081-bcc6-5feda1fef33a","Type":"ContainerDied","Data":"f9bbb713364098ef9fd72a2e91ef07de34f06735c33c0d494d06a61bb4ca1c70"}
Jul 16 22:46:00.420539 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.420373    2566 scope.go:117] "RemoveContainer" containerID="fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"
Jul 16 22:46:00.428934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.428907    2566 scope.go:117] "RemoveContainer" containerID="754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"
Jul 16 22:46:00.436720 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.436700    2566 scope.go:117] "RemoveContainer" containerID="fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"
Jul 16 22:46:00.437023 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:46:00.437003    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b\": container with ID starting with fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b not found: ID does not exist" containerID="fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"
Jul 16 22:46:00.437083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.437032    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b"} err="failed to get container status \"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b\": rpc error: code = NotFound desc = could not find container \"fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b\": container with ID starting with fb32814c0426df2715d639df53fff1b54d0db33dfd5b4df9966bcf9da62a581b not found: ID does not exist"
Jul 16 22:46:00.437083 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.437049    2566 scope.go:117] "RemoveContainer" containerID="754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"
Jul 16 22:46:00.437305 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:46:00.437286    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88\": container with ID starting with 754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88 not found: ID does not exist" containerID="754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"
Jul 16 22:46:00.437348 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.437313    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88"} err="failed to get container status \"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88\": rpc error: code = NotFound desc = could not find container \"754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88\": container with ID starting with 754cb848420f1f9de6362c0722de67097af212afbfbcdf9b06d6dc839c704e88 not found: ID does not exist"
Jul 16 22:46:00.443022 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.442998    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:46:00.448532 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:00.448508    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-cd554b854-6xg7k"]
Jul 16 22:46:02.319640 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:02.319600    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" path="/var/lib/kubelet/pods/6f6f6cc3-37ee-4081-bcc6-5feda1fef33a/volumes"
Jul 16 22:46:03.301242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301202    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"]
Jul 16 22:46:03.301540 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301527    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" containerName="main"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" containerName="main"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301554    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerName="main"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301559    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerName="main"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301568    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="storage-initializer"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301574    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="storage-initializer"
Jul 16 22:46:03.301585 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main"
Jul 16 22:46:03.301783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301592    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main"
Jul 16 22:46:03.301783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301659    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6f6cc3-37ee-4081-bcc6-5feda1fef33a" containerName="main"
Jul 16 22:46:03.301783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301666    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c809ae9-8950-4528-b1bc-bdd0dc790cd1" containerName="main"
Jul 16 22:46:03.301783 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.301674    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5f6e87-0e0c-4be5-9be2-a91adc8c512a" containerName="main"
Jul 16 22:46:03.306374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.306357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.318335 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.318304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"]
Jul 16 22:46:03.412520 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.412487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-cert\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.412520 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.412523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfrr\" (UniqueName: \"kubernetes.io/projected/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-kube-api-access-pjfrr\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.513479 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.513436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-cert\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.513479 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.513476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfrr\" (UniqueName: \"kubernetes.io/projected/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-kube-api-access-pjfrr\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.516152 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.516121    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-cert\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.522600 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.522571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfrr\" (UniqueName: \"kubernetes.io/projected/7f6c87ab-0d82-4b22-a6eb-94b681dd54ae-kube-api-access-pjfrr\") pod \"llmisvc-controller-manager-f8d5fb7c7-fmgtb\" (UID: \"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae\") " pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.617324 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.617230    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:03.758192 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.758164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"]
Jul 16 22:46:03.760650 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:46:03.760616    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7f6c87ab_0d82_4b22_a6eb_94b681dd54ae.slice/crio-09fa78cbee258296fc14cae8deaf13bbce3ef61d6d3034ec0759f6331d377a5d WatchSource:0}: Error finding container 09fa78cbee258296fc14cae8deaf13bbce3ef61d6d3034ec0759f6331d377a5d: Status 404 returned error can't find the container with id 09fa78cbee258296fc14cae8deaf13bbce3ef61d6d3034ec0759f6331d377a5d
Jul 16 22:46:03.762089 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:03.762071    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:46:04.440705 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:04.440663    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb" event={"ID":"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae","Type":"ContainerStarted","Data":"09fa78cbee258296fc14cae8deaf13bbce3ef61d6d3034ec0759f6331d377a5d"}
Jul 16 22:46:05.446598 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:05.446559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb" event={"ID":"7f6c87ab-0d82-4b22-a6eb-94b681dd54ae","Type":"ContainerStarted","Data":"89241ae4b12fd1371fe311c8f6366dde279ecc313cf754a7c2a6bf8ab6b59686"}
Jul 16 22:46:05.447028 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:05.446685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:05.466451 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:05.466385    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb" podStartSLOduration=1.7425545329999999 podStartE2EDuration="2.466369758s" totalImagesPullingTime="723.815225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:05.464095845 +0000 UTC m=+3365.559607163" watchObservedRunningTime="2026-07-16 22:46:05.466369758 +0000 UTC m=+3365.561881074"
Jul 16 22:46:36.454236 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.454208    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-f8d5fb7c7-fmgtb"
Jul 16 22:46:36.509792 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.509759    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 22:46:36.510002 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.509975    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerName="manager" containerID="cri-o://2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346" gracePeriod=30
Jul 16 22:46:36.764839 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.764789    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 22:46:36.907657 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.907616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") pod \"3cd417e6-878a-4e13-bef1-cafc23df6df6\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") "
Jul 16 22:46:36.907906 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.907736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp78n\" (UniqueName: \"kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n\") pod \"3cd417e6-878a-4e13-bef1-cafc23df6df6\" (UID: \"3cd417e6-878a-4e13-bef1-cafc23df6df6\") "
Jul 16 22:46:36.910019 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.909981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n" (OuterVolumeSpecName: "kube-api-access-gp78n") pod "3cd417e6-878a-4e13-bef1-cafc23df6df6" (UID: "3cd417e6-878a-4e13-bef1-cafc23df6df6"). InnerVolumeSpecName "kube-api-access-gp78n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:46:36.910019 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:36.910009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert" (OuterVolumeSpecName: "cert") pod "3cd417e6-878a-4e13-bef1-cafc23df6df6" (UID: "3cd417e6-878a-4e13-bef1-cafc23df6df6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:46:37.008680 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.008583    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp78n\" (UniqueName: \"kubernetes.io/projected/3cd417e6-878a-4e13-bef1-cafc23df6df6-kube-api-access-gp78n\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:46:37.008680 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.008616    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cd417e6-878a-4e13-bef1-cafc23df6df6-cert\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:46:37.592647 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.592608    2566 generic.go:356] "Generic (PLEG): container finished" podID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerID="2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346" exitCode=0
Jul 16 22:46:37.593123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.592656    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp"
Jul 16 22:46:37.593123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.592695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" event={"ID":"3cd417e6-878a-4e13-bef1-cafc23df6df6","Type":"ContainerDied","Data":"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"}
Jul 16 22:46:37.593123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.592744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" event={"ID":"3cd417e6-878a-4e13-bef1-cafc23df6df6","Type":"ContainerDied","Data":"56cfb2931995f322ee9f8703b77bf935cf55c772c7dc7655ae56e0a3365e68db"}
Jul 16 22:46:37.593123 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.592767    2566 scope.go:117] "RemoveContainer" containerID="2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"
Jul 16 22:46:37.603737 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.603718    2566 scope.go:117] "RemoveContainer" containerID="2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"
Jul 16 22:46:37.604025 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:46:37.604004    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346\": container with ID starting with 2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346 not found: ID does not exist" containerID="2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"
Jul 16 22:46:37.604122 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.604037    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346"} err="failed to get container status \"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346\": rpc error: code = NotFound desc = could not find container \"2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346\": container with ID starting with 2ff8771fc96ce4e012bf8aa1f277004d33fb96c7fcd0464175fbced078d30346 not found: ID does not exist"
Jul 16 22:46:37.613940 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.613920    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 22:46:37.618078 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:37.618058    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-78594f6654-2bbvp"]
Jul 16 22:46:38.319599 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:38.319565    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" path="/var/lib/kubelet/pods/3cd417e6-878a-4e13-bef1-cafc23df6df6/volumes"
Jul 16 22:46:41.758979 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:41.758936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-78594f6654-2bbvp" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.25:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:46:58.377013 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.376936    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:46:58.377358 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.377348    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerName="manager"
Jul 16 22:46:58.377399 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.377362    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerName="manager"
Jul 16 22:46:58.377434 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.377429    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd417e6-878a-4e13-bef1-cafc23df6df6" containerName="manager"
Jul 16 22:46:58.380441 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.380421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.384376 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.384348    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-l648x\""
Jul 16 22:46:58.384521 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.384395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 16 22:46:58.384521 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.384414    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:46:58.384521 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.384438    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:46:58.395537 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.395507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:46:58.485180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdktn\" (UniqueName: \"kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.485180 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.485443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.485443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.485443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.485443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.485399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586336 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdktn\" (UniqueName: \"kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586926 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.586926 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.587076 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.586962    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.589043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.589007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.589205 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.589188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.594827 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.594795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdktn\" (UniqueName: \"kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn\") pod \"rolling-upgrade-test-kserve-54bc76f55b-5nvsx\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.696081 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.695985    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:46:58.841876 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.841834    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:46:58.843804 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:46:58.843775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec4fa64_6a06_43b7_a22f_537c60fe1d77.slice/crio-aa262da314042d89102679dc5dbc9bf88bc26a6a35cbaa8019515af11ec291b2 WatchSource:0}: Error finding container aa262da314042d89102679dc5dbc9bf88bc26a6a35cbaa8019515af11ec291b2: Status 404 returned error can't find the container with id aa262da314042d89102679dc5dbc9bf88bc26a6a35cbaa8019515af11ec291b2
Jul 16 22:46:58.878049 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.878018    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:46:58.882402 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.882380    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:58.885352 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.885327    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-x8d7l\""
Jul 16 22:46:58.894796 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.894769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:46:58.990175 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.990137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlm78\" (UniqueName: \"kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:58.990348 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.990208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:58.990348 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:58.990235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.091278 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.091233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlm78\" (UniqueName: \"kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.091464 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.091294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.091464 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.091408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.091797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.091773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.093798 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.093781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.099802 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.099784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlm78\" (UniqueName: \"kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.195416 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.195367    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.349718 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.349687    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:46:59.353035 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:46:59.353001    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ba5445_111c_4f33_bae2_0ba80859882f.slice/crio-3dfb978f7d8bd03b82b244e6db6525a49fcfcb3c7eded8aadc8a5b35b91a7008 WatchSource:0}: Error finding container 3dfb978f7d8bd03b82b244e6db6525a49fcfcb3c7eded8aadc8a5b35b91a7008: Status 404 returned error can't find the container with id 3dfb978f7d8bd03b82b244e6db6525a49fcfcb3c7eded8aadc8a5b35b91a7008
Jul 16 22:46:59.700274 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.700134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" event={"ID":"6ec4fa64-6a06-43b7-a22f-537c60fe1d77","Type":"ContainerStarted","Data":"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"}
Jul 16 22:46:59.700274 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.700197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" event={"ID":"6ec4fa64-6a06-43b7-a22f-537c60fe1d77","Type":"ContainerStarted","Data":"aa262da314042d89102679dc5dbc9bf88bc26a6a35cbaa8019515af11ec291b2"}
Jul 16 22:46:59.702324 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.702293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" event={"ID":"51ba5445-111c-4f33-bae2-0ba80859882f","Type":"ContainerStarted","Data":"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"}
Jul 16 22:46:59.702473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.702344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" event={"ID":"51ba5445-111c-4f33-bae2-0ba80859882f","Type":"ContainerStarted","Data":"3dfb978f7d8bd03b82b244e6db6525a49fcfcb3c7eded8aadc8a5b35b91a7008"}
Jul 16 22:46:59.702473 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.702389    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:46:59.719102 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.719044    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" podStartSLOduration=1.719029 podStartE2EDuration="1.719029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:59.717125425 +0000 UTC m=+3419.812636754" watchObservedRunningTime="2026-07-16 22:46:59.719029 +0000 UTC m=+3419.814540317"
Jul 16 22:46:59.735407 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:46:59.735349    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" podStartSLOduration=1.735330116 podStartE2EDuration="1.735330116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:59.735205442 +0000 UTC m=+3419.830716759" watchObservedRunningTime="2026-07-16 22:46:59.735330116 +0000 UTC m=+3419.830841435"
Jul 16 22:47:00.696719 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:00.696676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:08.696374 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:08.696327    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:08.708609 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:08.708575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:08.757654 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:08.757619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:30.709934 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:30.709904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:47:42.307728 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:42.307694    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:47:42.494064 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:42.494024    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:47:50.710068 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:50.710030    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 22:47:57.335450 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.335408    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" podUID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" containerName="main" containerID="cri-o://9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5" gracePeriod=45
Jul 16 22:47:57.516254 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.516213    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" containerName="main" containerID="cri-o://44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b" gracePeriod=45
Jul 16 22:47:57.641997 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.641971    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdktn\" (UniqueName: \"kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690562    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690619    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.690788 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home\") pod \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\" (UID: \"6ec4fa64-6a06-43b7-a22f-537c60fe1d77\") "
Jul 16 22:47:57.691238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.690806    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:57.691238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.691004    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tmp-dir\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.691238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.691046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:57.691238 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.691063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home" (OuterVolumeSpecName: "home") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:57.696659 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.696628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm" (OuterVolumeSpecName: "dshm") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:57.696797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.696657    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn" (OuterVolumeSpecName: "kube-api-access-kdktn") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "kube-api-access-kdktn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:47:57.696797 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.696769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ec4fa64-6a06-43b7-a22f-537c60fe1d77" (UID: "6ec4fa64-6a06-43b7-a22f-537c60fe1d77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:47:57.755751 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.755726    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:47:57.791443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791409    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds\") pod \"51ba5445-111c-4f33-bae2-0ba80859882f\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") "
Jul 16 22:47:57.791630 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs\") pod \"51ba5445-111c-4f33-bae2-0ba80859882f\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") "
Jul 16 22:47:57.791630 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlm78\" (UniqueName: \"kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78\") pod \"51ba5445-111c-4f33-bae2-0ba80859882f\" (UID: \"51ba5445-111c-4f33-bae2-0ba80859882f\") "
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791671    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-model-cache\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791685    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-dshm\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791694    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-home\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791702    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791698    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "51ba5445-111c-4f33-bae2-0ba80859882f" (UID: "51ba5445-111c-4f33-bae2-0ba80859882f"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:47:57.791763 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.791712    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdktn\" (UniqueName: \"kubernetes.io/projected/6ec4fa64-6a06-43b7-a22f-537c60fe1d77-kube-api-access-kdktn\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.793838 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.793790    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78" (OuterVolumeSpecName: "kube-api-access-wlm78") pod "51ba5445-111c-4f33-bae2-0ba80859882f" (UID: "51ba5445-111c-4f33-bae2-0ba80859882f"). InnerVolumeSpecName "kube-api-access-wlm78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:47:57.793933 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.793845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51ba5445-111c-4f33-bae2-0ba80859882f" (UID: "51ba5445-111c-4f33-bae2-0ba80859882f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:47:57.892297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.892208    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51ba5445-111c-4f33-bae2-0ba80859882f-tls-certs\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.892297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.892237    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlm78\" (UniqueName: \"kubernetes.io/projected/51ba5445-111c-4f33-bae2-0ba80859882f-kube-api-access-wlm78\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.892297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.892248    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/51ba5445-111c-4f33-bae2-0ba80859882f-tokenizer-uds\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:47:57.984916 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.984881    2566 generic.go:356] "Generic (PLEG): container finished" podID="51ba5445-111c-4f33-bae2-0ba80859882f" containerID="44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b" exitCode=0
Jul 16 22:47:57.985096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.984931    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"
Jul 16 22:47:57.985096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.984964    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" event={"ID":"51ba5445-111c-4f33-bae2-0ba80859882f","Type":"ContainerDied","Data":"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"}
Jul 16 22:47:57.985096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.985016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27" event={"ID":"51ba5445-111c-4f33-bae2-0ba80859882f","Type":"ContainerDied","Data":"3dfb978f7d8bd03b82b244e6db6525a49fcfcb3c7eded8aadc8a5b35b91a7008"}
Jul 16 22:47:57.985096 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.985037    2566 scope.go:117] "RemoveContainer" containerID="44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"
Jul 16 22:47:57.986449 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.986426    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" containerID="9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5" exitCode=0
Jul 16 22:47:57.986559 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.986478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"
Jul 16 22:47:57.986618 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.986480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" event={"ID":"6ec4fa64-6a06-43b7-a22f-537c60fe1d77","Type":"ContainerDied","Data":"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"}
Jul 16 22:47:57.986667 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.986606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx" event={"ID":"6ec4fa64-6a06-43b7-a22f-537c60fe1d77","Type":"ContainerDied","Data":"aa262da314042d89102679dc5dbc9bf88bc26a6a35cbaa8019515af11ec291b2"}
Jul 16 22:47:57.995226 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.995209    2566 scope.go:117] "RemoveContainer" containerID="44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"
Jul 16 22:47:57.995529 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:47:57.995507    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b\": container with ID starting with 44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b not found: ID does not exist" containerID="44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"
Jul 16 22:47:57.995652 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.995541    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b"} err="failed to get container status \"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b\": rpc error: code = NotFound desc = could not find container \"44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b\": container with ID starting with 44a8d0fd98ae9df8a92d525cd1d27a7a588bec2f2a0f3d75633d07fb8868297b not found: ID does not exist"
Jul 16 22:47:57.995652 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:57.995564    2566 scope.go:117] "RemoveContainer" containerID="9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"
Jul 16 22:47:58.003978 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.003958    2566 scope.go:117] "RemoveContainer" containerID="9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"
Jul 16 22:47:58.005653 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:47:58.005627    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5\": container with ID starting with 9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5 not found: ID does not exist" containerID="9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"
Jul 16 22:47:58.005752 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.005661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5"} err="failed to get container status \"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5\": rpc error: code = NotFound desc = could not find container \"9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5\": container with ID starting with 9fd1ee8f788144d03ac40ed02550b8b2553a17fb9f992ee4cfa6f938f57b98f5 not found: ID does not exist"
Jul 16 22:47:58.011765 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.011744    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:47:58.014242 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.014221    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-mdh27"]
Jul 16 22:47:58.026644 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.026616    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:47:58.031037 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.031015    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-5nvsx"]
Jul 16 22:47:58.319936 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.319897    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" path="/var/lib/kubelet/pods/51ba5445-111c-4f33-bae2-0ba80859882f/volumes"
Jul 16 22:47:58.320394 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:47:58.320375    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" path="/var/lib/kubelet/pods/6ec4fa64-6a06-43b7-a22f-537c60fe1d77/volumes"
Jul 16 22:48:06.313921 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.313886    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ghvv6/must-gather-nrmqp"]
Jul 16 22:48:06.314297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314219    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" containerName="main"
Jul 16 22:48:06.314297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314229    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" containerName="main"
Jul 16 22:48:06.314297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314240    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" containerName="main"
Jul 16 22:48:06.314297 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314247    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" containerName="main"
Jul 16 22:48:06.314443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314305    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ba5445-111c-4f33-bae2-0ba80859882f" containerName="main"
Jul 16 22:48:06.314443 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.314313    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec4fa64-6a06-43b7-a22f-537c60fe1d77" containerName="main"
Jul 16 22:48:06.319089 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.319065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.326410 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.326324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghvv6\"/\"kube-root-ca.crt\""
Jul 16 22:48:06.326556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.326532    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ghvv6\"/\"openshift-service-ca.crt\""
Jul 16 22:48:06.343401 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.343370    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghvv6/must-gather-nrmqp"]
Jul 16 22:48:06.368902 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.368874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.369043 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.368920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dclkz\" (UniqueName: \"kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.469943 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.469914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.470112 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.469962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dclkz\" (UniqueName: \"kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.470269 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.470252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.483360 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.483331    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dclkz\" (UniqueName: \"kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz\") pod \"must-gather-nrmqp\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") " pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.628747 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.628660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:06.762127 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:06.762105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ghvv6/must-gather-nrmqp"]
Jul 16 22:48:06.763991 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:48:06.763955    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd54630a7_7b21_42d6_ba80_9f971326f045.slice/crio-d4898229f03d2e5ed6f6b814addfa061d4f7234a3f5e0ede221842d5848340ed WatchSource:0}: Error finding container d4898229f03d2e5ed6f6b814addfa061d4f7234a3f5e0ede221842d5848340ed: Status 404 returned error can't find the container with id d4898229f03d2e5ed6f6b814addfa061d4f7234a3f5e0ede221842d5848340ed
Jul 16 22:48:07.038163 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:07.038128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" event={"ID":"d54630a7-7b21-42d6-ba80-9f971326f045","Type":"ContainerStarted","Data":"d4898229f03d2e5ed6f6b814addfa061d4f7234a3f5e0ede221842d5848340ed"}
Jul 16 22:48:14.083650 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:14.083601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" event={"ID":"d54630a7-7b21-42d6-ba80-9f971326f045","Type":"ContainerStarted","Data":"37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9"}
Jul 16 22:48:14.084272 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:14.083661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" event={"ID":"d54630a7-7b21-42d6-ba80-9f971326f045","Type":"ContainerStarted","Data":"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"}
Jul 16 22:48:14.110869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:14.110792    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" podStartSLOduration=1.754980118 podStartE2EDuration="8.110777469s" totalImagesPullingTime="6.355797351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:14.10933459 +0000 UTC m=+3494.204845908" watchObservedRunningTime="2026-07-16 22:48:14.110777469 +0000 UTC m=+3494.206288833"
Jul 16 22:48:22.282917 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:22.282842    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:23.833044 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:23.833017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:25.343706 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:25.343678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:26.779245 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:26.779220    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:28.221390 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:28.221363    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:29.693437 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:29.693411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:31.110485 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:31.110430    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:32.558195 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:32.558168    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:34.007970 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:34.007946    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:35.528360 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:35.528333    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:37.297042 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:37.297015    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:38.731342 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:38.731308    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:40.224631 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:40.224606    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:41.659008 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:41.658974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:43.084869 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:43.084843    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:44.517917 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:44.517889    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-w44vk_5f8ff498-5f38-437c-aa39-02e8bc124806/storage-initializer/0.log"
Jul 16 22:48:49.155662 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:49.155631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-w4lkk_c3de1e90-90b7-4d7b-8189-afbe3e742cb4/authorino/0.log"
Jul 16 22:48:50.274600 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:50.274565    2566 generic.go:356] "Generic (PLEG): container finished" podID="d54630a7-7b21-42d6-ba80-9f971326f045" containerID="1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac" exitCode=0
Jul 16 22:48:50.275016 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:50.274610    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" event={"ID":"d54630a7-7b21-42d6-ba80-9f971326f045","Type":"ContainerDied","Data":"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"}
Jul 16 22:48:50.275016 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:50.274951    2566 scope.go:117] "RemoveContainer" containerID="1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"
Jul 16 22:48:51.244947 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:51.244915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ghvv6_must-gather-nrmqp_d54630a7-7b21-42d6-ba80-9f971326f045/gather/0.log"
Jul 16 22:48:54.924299 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:54.924235    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wjrxm_89297fd4-1e1b-42b3-aaec-182a16f6a4b9/global-pull-secret-syncer/0.log"
Jul 16 22:48:55.023758 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:55.023728    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kkkpd_936c198b-a37a-4426-bc7d-9cbcd336282d/konnectivity-agent/0.log"
Jul 16 22:48:55.155806 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:55.155774    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-100.ec2.internal_dc363aecea93ceaa1e0b8c8b651a90ec/haproxy/0.log"
Jul 16 22:48:56.876319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:56.876276    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-ghvv6/must-gather-nrmqp"]
Jul 16 22:48:56.876719 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:56.876495    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-ghvv6/must-gather-nrmqp" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="copy" containerID="cri-o://37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9" gracePeriod=2
Jul 16 22:48:56.879367 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:56.879336    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrmqp\" is forbidden: User \"system:node:ip-10-0-143-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ghvv6\": no relationship found between node 'ip-10-0-143-100.ec2.internal' and this object" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:56.880946 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:56.880486    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-ghvv6/must-gather-nrmqp"]
Jul 16 22:48:57.113929 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.113906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ghvv6_must-gather-nrmqp_d54630a7-7b21-42d6-ba80-9f971326f045/copy/0.log"
Jul 16 22:48:57.116559 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.116542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:57.118865 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:57.118810    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrmqp\" is forbidden: User \"system:node:ip-10-0-143-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ghvv6\": no relationship found between node 'ip-10-0-143-100.ec2.internal' and this object" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:57.249552 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.249523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output\") pod \"d54630a7-7b21-42d6-ba80-9f971326f045\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") "
Jul 16 22:48:57.249729 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.249563    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dclkz\" (UniqueName: \"kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz\") pod \"d54630a7-7b21-42d6-ba80-9f971326f045\" (UID: \"d54630a7-7b21-42d6-ba80-9f971326f045\") "
Jul 16 22:48:57.251933 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.251904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz" (OuterVolumeSpecName: "kube-api-access-dclkz") pod "d54630a7-7b21-42d6-ba80-9f971326f045" (UID: "d54630a7-7b21-42d6-ba80-9f971326f045"). InnerVolumeSpecName "kube-api-access-dclkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:57.256183 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.256155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "d54630a7-7b21-42d6-ba80-9f971326f045" (UID: "d54630a7-7b21-42d6-ba80-9f971326f045"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:57.308704 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.308674    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-ghvv6_must-gather-nrmqp_d54630a7-7b21-42d6-ba80-9f971326f045/copy/0.log"
Jul 16 22:48:57.309023 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.308998    2566 generic.go:356] "Generic (PLEG): container finished" podID="d54630a7-7b21-42d6-ba80-9f971326f045" containerID="37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9" exitCode=143
Jul 16 22:48:57.309088 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.309050    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:57.309088 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.309066    2566 scope.go:117] "RemoveContainer" containerID="37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9"
Jul 16 22:48:57.311514 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:57.311485    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrmqp\" is forbidden: User \"system:node:ip-10-0-143-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ghvv6\": no relationship found between node 'ip-10-0-143-100.ec2.internal' and this object" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:57.317236 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.317214    2566 scope.go:117] "RemoveContainer" containerID="1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"
Jul 16 22:48:57.319670 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:57.319641    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrmqp\" is forbidden: User \"system:node:ip-10-0-143-100.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-ghvv6\": no relationship found between node 'ip-10-0-143-100.ec2.internal' and this object" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" pod="openshift-must-gather-ghvv6/must-gather-nrmqp"
Jul 16 22:48:57.329933 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.329902    2566 scope.go:117] "RemoveContainer" containerID="37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9"
Jul 16 22:48:57.330232 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:57.330214    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9\": container with ID starting with 37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9 not found: ID does not exist" containerID="37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9"
Jul 16 22:48:57.330282 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.330244    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9"} err="failed to get container status \"37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9\": rpc error: code = NotFound desc = could not find container \"37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9\": container with ID starting with 37c9aa70f88e5df8db638c135b8239ad2c4c8a308b45828ef6a49a8b047c51a9 not found: ID does not exist"
Jul 16 22:48:57.330282 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.330262    2566 scope.go:117] "RemoveContainer" containerID="1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"
Jul 16 22:48:57.330508 ip-10-0-143-100 kubenswrapper[2566]: E0716 22:48:57.330491    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac\": container with ID starting with 1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac not found: ID does not exist" containerID="1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"
Jul 16 22:48:57.330556 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.330512    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac"} err="failed to get container status \"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac\": rpc error: code = NotFound desc = could not find container \"1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac\": container with ID starting with 1267730cdfad82fdfdcdab85c979b3bb19302fe14cef3822959ed040f9543bac not found: ID does not exist"
Jul 16 22:48:57.350497 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.350472    2566 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d54630a7-7b21-42d6-ba80-9f971326f045-must-gather-output\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:48:57.350497 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:57.350496    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dclkz\" (UniqueName: \"kubernetes.io/projected/d54630a7-7b21-42d6-ba80-9f971326f045-kube-api-access-dclkz\") on node \"ip-10-0-143-100.ec2.internal\" DevicePath \"\""
Jul 16 22:48:58.318966 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:48:58.318932    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" path="/var/lib/kubelet/pods/d54630a7-7b21-42d6-ba80-9f971326f045/volumes"
Jul 16 22:49:00.074285 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:00.074245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-w4lkk_c3de1e90-90b7-4d7b-8189-afbe3e742cb4/authorino/0.log"
Jul 16 22:49:02.185956 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.185928    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-d4stg_51dceddc-d757-42c7-aecc-15ca5648ff0c/kube-state-metrics/0.log"
Jul 16 22:49:02.212619 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.212591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-d4stg_51dceddc-d757-42c7-aecc-15ca5648ff0c/kube-rbac-proxy-main/0.log"
Jul 16 22:49:02.248067 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.248038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-d4stg_51dceddc-d757-42c7-aecc-15ca5648ff0c/kube-rbac-proxy-self/0.log"
Jul 16 22:49:02.557682 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.557658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsx9k_30f239df-f578-4e0e-90a0-eebbdcd81636/node-exporter/0.log"
Jul 16 22:49:02.584937 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.584910    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsx9k_30f239df-f578-4e0e-90a0-eebbdcd81636/kube-rbac-proxy/0.log"
Jul 16 22:49:02.612665 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.612647    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hsx9k_30f239df-f578-4e0e-90a0-eebbdcd81636/init-textfile/0.log"
Jul 16 22:49:02.953354 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.953325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-x9pg7_8c605f47-1e34-4980-aeb7-f3999b6492d9/prometheus-operator/0.log"
Jul 16 22:49:02.974525 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:02.974496    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-x9pg7_8c605f47-1e34-4980-aeb7-f3999b6492d9/kube-rbac-proxy/0.log"
Jul 16 22:49:03.000493 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.000467    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-8fgw4_14fec963-1510-4b5b-be31-706b04c39634/prometheus-operator-admission-webhook/0.log"
Jul 16 22:49:03.123113 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.123083    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/thanos-query/0.log"
Jul 16 22:49:03.150203 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.150174    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/kube-rbac-proxy-web/0.log"
Jul 16 22:49:03.174435 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.174404    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/kube-rbac-proxy/0.log"
Jul 16 22:49:03.199375 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.199354    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/prom-label-proxy/0.log"
Jul 16 22:49:03.221540 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.221472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/kube-rbac-proxy-rules/0.log"
Jul 16 22:49:03.245087 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:03.245062    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-59b9bd979b-kffjz_f9dc45a7-7a38-466e-befc-08d99086674a/kube-rbac-proxy-metrics/0.log"
Jul 16 22:49:04.304228 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.304195    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-krmrx_9fa92db6-85f8-4b66-aa03-7c8fa2457c73/networking-console-plugin/0.log"
Jul 16 22:49:04.796635 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.796603    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"]
Jul 16 22:49:04.797031 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797004    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="gather"
Jul 16 22:49:04.797031 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797026    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="gather"
Jul 16 22:49:04.797204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797054    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="copy"
Jul 16 22:49:04.797204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797061    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="copy"
Jul 16 22:49:04.797204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797142    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="gather"
Jul 16 22:49:04.797204 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.797153    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54630a7-7b21-42d6-ba80-9f971326f045" containerName="copy"
Jul 16 22:49:04.803622 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.803602    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:04.806405 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.806384    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v9ch\"/\"openshift-service-ca.crt\""
Jul 16 22:49:04.807319 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.807303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5v9ch\"/\"default-dockercfg-9x8pv\""
Jul 16 22:49:04.807366 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.807317    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v9ch\"/\"kube-root-ca.crt\""
Jul 16 22:49:04.811505 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.811486    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"]
Jul 16 22:49:04.913545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.913507    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-proc\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:04.913731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.913577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-sys\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:04.913731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.913595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-lib-modules\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:04.913731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.913618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87l7m\" (UniqueName: \"kubernetes.io/projected/23abeade-237a-4a36-92bc-806943568bb4-kube-api-access-87l7m\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:04.913731 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:04.913698    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-podres\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014600 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014555    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-proc\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-sys\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-lib-modules\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87l7m\" (UniqueName: \"kubernetes.io/projected/23abeade-237a-4a36-92bc-806943568bb4-kube-api-access-87l7m\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014688    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-proc\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.014801 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-sys\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.015062 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-podres\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.015062 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-lib-modules\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.015062 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.014963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23abeade-237a-4a36-92bc-806943568bb4-podres\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.023046 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.023021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87l7m\" (UniqueName: \"kubernetes.io/projected/23abeade-237a-4a36-92bc-806943568bb4-kube-api-access-87l7m\") pod \"perf-node-gather-daemonset-swcxv\" (UID: \"23abeade-237a-4a36-92bc-806943568bb4\") " pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.115172 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.115078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.256623 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.256591    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"]
Jul 16 22:49:05.259952 ip-10-0-143-100 kubenswrapper[2566]: W0716 22:49:05.259921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23abeade_237a_4a36_92bc_806943568bb4.slice/crio-31c3b52940782fda4cab8ffc81da993ad6d0a6907b508071ba4227143d0b9ac1 WatchSource:0}: Error finding container 31c3b52940782fda4cab8ffc81da993ad6d0a6907b508071ba4227143d0b9ac1: Status 404 returned error can't find the container with id 31c3b52940782fda4cab8ffc81da993ad6d0a6907b508071ba4227143d0b9ac1
Jul 16 22:49:05.305361 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.305337    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79c4bccd4b-9kzkn_2325992e-3418-4166-9578-9c2dfce4e668/console/0.log"
Jul 16 22:49:05.349587 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.349560    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv" event={"ID":"23abeade-237a-4a36-92bc-806943568bb4","Type":"ContainerStarted","Data":"ef6bd0e1da433de16b9675d23f75b00a633280b85b60c7875e559ef21617549d"}
Jul 16 22:49:05.349739 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.349607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv" event={"ID":"23abeade-237a-4a36-92bc-806943568bb4","Type":"ContainerStarted","Data":"31c3b52940782fda4cab8ffc81da993ad6d0a6907b508071ba4227143d0b9ac1"}
Jul 16 22:49:05.349739 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.349674    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:05.388397 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.388291    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv" podStartSLOduration=1.388275368 podStartE2EDuration="1.388275368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:49:05.387210306 +0000 UTC m=+3545.482721623" watchObservedRunningTime="2026-07-16 22:49:05.388275368 +0000 UTC m=+3545.483786684"
Jul 16 22:49:05.392417 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:05.392384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-z98vj_bfa7f7d4-d2e4-4dfc-93f6-fedb5d560687/download-server/0.log"
Jul 16 22:49:06.908111 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:06.908084    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4cx2_013a28d9-1af6-4734-a6bb-45ce6753a567/dns/0.log"
Jul 16 22:49:06.930387 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:06.930361    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c4cx2_013a28d9-1af6-4734-a6bb-45ce6753a567/kube-rbac-proxy/0.log"
Jul 16 22:49:07.085385 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:07.085359    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9mc6r_6ea52686-3288-4855-aef7-2ba5ac429dbf/dns-node-resolver/0.log"
Jul 16 22:49:07.661200 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:07.661168    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5fcb87f568-6gz4t_eb625f60-4985-418b-ac4e-241c97dd45c6/registry/0.log"
Jul 16 22:49:07.715571 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:07.715545    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fs66g_e1304cb1-326c-4c14-8464-201068718829/node-ca/0.log"
Jul 16 22:49:09.987545 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:09.987519    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq788_c35f0b32-6ae6-4250-b447-9c1ab70b5235/serve-healthcheck-canary/0.log"
Jul 16 22:49:10.546383 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:10.546339    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9b47_7a7e0641-b4ef-4104-8a71-b9795f4c6340/kube-rbac-proxy/0.log"
Jul 16 22:49:10.568007 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:10.567979    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9b47_7a7e0641-b4ef-4104-8a71-b9795f4c6340/exporter/0.log"
Jul 16 22:49:10.590298 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:10.590277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p9b47_7a7e0641-b4ef-4104-8a71-b9795f4c6340/extractor/0.log"
Jul 16 22:49:11.364077 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:11.364051    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5v9ch/perf-node-gather-daemonset-swcxv"
Jul 16 22:49:13.550370 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:13.550338    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5fcdcd6776-99jlx_742b2f6e-8d01-4e3c-9825-6673d123e697/manager/0.log"
Jul 16 22:49:14.378919 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:14.378890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-f8d5fb7c7-fmgtb_7f6c87ab-0d82-4b22-a6eb-94b681dd54ae/manager/0.log"
Jul 16 22:49:14.402287 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:14.402262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wsrth_33f9d154-fd1f-45d9-a7a9-ebbd9709be15/server/0.log"
Jul 16 22:49:15.412405 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:15.412377    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cf58t_99fa3c05-0f9a-496b-9110-7baf5c38c04e/s3-init/0.log"
Jul 16 22:49:15.429141 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:15.429121    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cf58t_99fa3c05-0f9a-496b-9110-7baf5c38c04e/download-hf-model/0.log"
Jul 16 22:49:22.106566 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.106537    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/kube-multus-additional-cni-plugins/0.log"
Jul 16 22:49:22.130889 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.130863    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/egress-router-binary-copy/0.log"
Jul 16 22:49:22.152278 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.152252    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/cni-plugins/0.log"
Jul 16 22:49:22.173641 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.173616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/bond-cni-plugin/0.log"
Jul 16 22:49:22.195741 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.195720    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/routeoverride-cni/0.log"
Jul 16 22:49:22.216581 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.216564    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/whereabouts-cni-bincopy/0.log"
Jul 16 22:49:22.238569 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.238542    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c899z_10098757-a07b-47c0-9786-61b5704881b0/whereabouts-cni/0.log"
Jul 16 22:49:22.687001 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.686973    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zm8gq_924fa652-0d0a-4d38-ae73-d20387be6f26/kube-multus/0.log"
Jul 16 22:49:22.755806 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.755776    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jsh7v_fdc51d15-a2f4-40e6-b973-11dd4094fe35/network-metrics-daemon/0.log"
Jul 16 22:49:22.775024 ip-10-0-143-100 kubenswrapper[2566]: I0716 22:49:22.774999    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jsh7v_fdc51d15-a2f4-40e6-b973-11dd4094fe35/kube-rbac-proxy/0.log"