Jul 31 14:46:06.750409 ip-10-0-137-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 14:46:06.750421 ip-10-0-137-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 14:46:06.750429 ip-10-0-137-96 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 14:46:06.750654 ip-10-0-137-96 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 14:46:16.702682 ip-10-0-137-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 14:46:16.702695 ip-10-0-137-96 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c9c2fefcfb864ad58747595195de19d9 --
Jul 31 14:47:57.748594 ip-10-0-137-96 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:47:58.048326 ip-10-0-137-96 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:58.048326 ip-10-0-137-96 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:47:58.048326 ip-10-0-137-96 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:58.048326 ip-10-0-137-96 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:47:58.048326 ip-10-0-137-96 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:47:58.049665 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.049578    2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:47:58.052886 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052871    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.052886 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052886    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052890    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052894    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052897    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052902    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052905    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052908    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052911    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052914    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052935    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052939    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052942    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052944    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052947    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052949    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052952    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052954    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052957    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052960    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.052955 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052963    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052966    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052969    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052971    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052974    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052976    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052979    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052981    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052984    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052986    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052989    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052991    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052993    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052996    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.052998    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053001    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053003    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053006    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053008    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053011    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.053481 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053014    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053016    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053019    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053022    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053025    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053027    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053030    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053034    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053036    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053039    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053043    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053046    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053049    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053053    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053057    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053060    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053063    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053065    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053068    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.053954 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053071    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053073    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053076    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053078    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053081    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053083    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053085    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053088    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053091    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053094    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053096    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053099    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053101    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053104    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053106    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053108    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053114    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053117    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053120    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.054437 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053122    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053125    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053127    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053130    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053133    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053136    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053138    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053140    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053147    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053150    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053153    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053156    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053159    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053162    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053166    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053169    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053171    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053174    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053176    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053179    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.054894 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053181    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053870    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053877    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053880    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053883    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053885    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053888    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053891    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053893    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053896    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053899    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053901    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053904    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053907    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053909    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053912    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053914    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053917    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053920    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053922    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.055400 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053925    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053928    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053931    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053933    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053936    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053939    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053941    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053944    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053946    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053949    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053951    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053954    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053957    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053960    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053964    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053968    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053971    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053974    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053977    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053979    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.055895 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053982    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053984    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053987    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053991    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053995    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.053998    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054000    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054003    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054006    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054009    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054011    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054014    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054017    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054020    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054022    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054025    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054027    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054030    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054032    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.056407 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054035    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054038    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054041    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054044    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054047    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054050    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054053    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054056    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054059    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054062    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054064    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054067    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054070    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054073    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054076    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054078    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054081    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054084    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054087    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.056891 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054090    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054092    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054094    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054098    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054100    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054103    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054106    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054109    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054112    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054115    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054117    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054120    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054123    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054125    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054127    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054130    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054133    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054136    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054138    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054140    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.057394 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054143    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054146    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054537    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054542    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054545    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054548    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054550    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054553    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054555    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054558    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054561    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054564    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054567    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054569    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054572    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054574    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054577    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054581    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054586    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.057883 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054589    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054592    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054595    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054598    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054600    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054603    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054605    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054608    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054610    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054613    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054616    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054619    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054623    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054626    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054629    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054631    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054634    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054636    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054639    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054641    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.058362 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054644    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054647    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054649    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054652    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054655    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054657    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054660    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054663    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054665    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054668    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054670    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054673    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054676    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054678    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054681    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054683    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054686    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054689    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054691    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054694    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.058884 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054696    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054699    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054701    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054704    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054707    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054709    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054713    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054716    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054719    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054722    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054724    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054727    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054730    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054733    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054736    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054738    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054741    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054744    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054747    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054750    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.059409 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054752    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054755    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054758    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054765    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054767    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054770    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054773    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054776    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054779    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054782    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054785    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054788    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054791    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054793    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054796    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054798    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054801    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054803    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054806    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054808    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.059892 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054811    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.054813    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054892    2557 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054899    2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054905    2557 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054910    2557 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054914    2557 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054917    2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054922    2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054926    2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054930    2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054933    2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054936    2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054940    2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054943    2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054948    2557 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054951    2557 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054954    2557 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054957    2557 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054960    2557 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054965    2557 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054968    2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054971    2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054974    2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:47:58.060421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054977    2557 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054981    2557 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054984    2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054987    2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054990    2557 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054993    2557 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054996    2557 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.054999    2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055002    2557 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055005    2557 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055009    2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055012    2557 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055015    2557 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055018    2557 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055021    2557 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055024    2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055028    2557 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055031    2557 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055034    2557 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055038    2557 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055041    2557 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055046    2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055049    2557 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055053    2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055056    2557 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:47:58.061016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055059    2557 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055062    2557 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055065    2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055068    2557 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055071    2557 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055074    2557 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055077    2557 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055081    2557 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055084    2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055088    2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055091    2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055094    2557 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055097    2557 flags.go:64] FLAG: --help="false"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055100    2557 flags.go:64] FLAG: --hostname-override="ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055104    2557 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055107    2557 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055110    2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055121    2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055126    2557 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055129    2557 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055131    2557 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055134    2557 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055137    2557 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:47:58.061662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055140    2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055143    2557 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055146    2557 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055149    2557 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055153    2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055157    2557 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055160    2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055164    2557 flags.go:64] FLAG: --lock-file=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055166    2557 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055170    2557 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055173    2557 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055178    2557 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055181    2557 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055184    2557 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055187    2557 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055189    2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055193    2557 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055196    2557 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055199    2557 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055203    2557 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055206    2557 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055210    2557 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055213    2557 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055216    2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055219    2557 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:47:58.062262 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055222    2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055225    2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055228    2557 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055231    2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055250    2557 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055254    2557 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055257    2557 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055259    2557 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055262    2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055268    2557 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055271    2557 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055274    2557 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055279    2557 flags.go:64] FLAG: --port="10250"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055282    2557 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055285    2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff95446014d2cc32"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055291    2557 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055294    2557 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055297    2557 flags.go:64] FLAG: --register-node="true"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055299    2557 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055303    2557 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055306    2557 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055309    2557 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055312    2557 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055316    2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055319    2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:47:58.062883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055322    2557 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055325    2557 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055327    2557 flags.go:64] FLAG: --runonce="false"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055330    2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055333    2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055336    2557 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055339    2557 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055342    2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055344    2557 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055347    2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055350    2557 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055353    2557 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055356    2557 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055358    2557 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055361    2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055364    2557 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055367    2557 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055370    2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055375    2557 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055379    2557 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055383    2557 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055387    2557 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055392    2557 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055394    2557 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055397    2557 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:47:58.063598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055400    2557 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055403    2557 flags.go:64] FLAG: --v="2"
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055407    2557 flags.go:64] FLAG: --version="false"
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055411    2557 flags.go:64] FLAG: --vmodule=""
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055415    2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.055418    2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055559    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055563    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055566    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055568    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055571    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055574    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055576    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055579    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055582    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055586    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055588    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055591    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055593    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055596    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055598    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.064230 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055601    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055603    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055606    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055608    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055613    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055618    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055621    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055624    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055627    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055630    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055633    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055635    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055638    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055640    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055643    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055645    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055648    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055650    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055653    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.064759 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055655    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055658    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055661    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055663    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055665    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055668    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055670    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055673    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055675    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055678    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055680    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055683    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055685    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055687    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055690    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055692    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055695    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055697    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055701    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055703    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.065228 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055706    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055709    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055712    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055715    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055718    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055720    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055723    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055725    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055728    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055731    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055733    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055736    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055738    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055741    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055743    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055746    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055748    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055751    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055754    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055756    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.065769 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055759    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055762    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055764    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055766    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055770    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055773    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055776    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055780    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055783    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055786    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055790    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055793    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055795    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055798    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055801    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055804    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055806    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055809    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055811    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.066287 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055814    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055816    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055819    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055821    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055823    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.055826    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.056408    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.063164    2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.063180    2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.063834    2557 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.063842    2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063913    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063919    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063923    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063926    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.066750 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063929    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063932    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063935    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063938    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063941    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063944    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063947    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063950    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063953    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063956    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063958    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063961    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063969    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063972    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063975    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063978    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063981    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063983    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063987    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063990    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.067129 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063992    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063995    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.063997    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064000    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064003    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064006    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064009    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064011    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064014    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064017    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064019    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064022    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064024    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064028    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064032    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064035    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064038    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064041    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064043    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.067696 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064046    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064048    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064051    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064053    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064057    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064061    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064069    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064081    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064084    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064086    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064089    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064092    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064094    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064098    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064101    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064103    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064106    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064108    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064112    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.068175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064114    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064117    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064120    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064122    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064125    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064127    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064130    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064133    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064135    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064139    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064143    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064146    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064149    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064151    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064154    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064156    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064159    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064162    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064164    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.068668 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064167    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064175    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064178    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064180    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064183    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064185    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064188    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064190    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064193    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064196    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064198    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064201    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064204    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064206    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064209    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064211    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064214    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.069143 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064217    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.064222    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064381    2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064386    2557 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064389    2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064392    2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064395    2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064399    2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064402    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064405    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064408    2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064411    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064413    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064416    2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:47:58.069596 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064419    2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064421    2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064424    2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064432    2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064435    2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064437    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064440    2557 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064443    2557 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064446    2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064448    2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064451    2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064453    2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064456    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064459    2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064462    2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064464    2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064467    2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064469    2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064472    2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:47:58.069951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064475    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064477    2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064479    2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064482    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064485    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064487    2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064491    2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064495    2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064499    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064502    2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064505    2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064507    2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064510    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064512    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064515    2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064517    2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064520    2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064528    2557 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064530    2557 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:47:58.070679 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064534    2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064536    2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064539    2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064541    2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064544    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064546    2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064549    2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064552    2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064555    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064557    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064560    2557 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064562    2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064565    2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064567    2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064570    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064572    2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064575    2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064577    2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064580    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064582    2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:47:58.071431 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064584    2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064587    2557 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064589    2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064592    2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064594    2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064596    2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064599    2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064601    2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064604    2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064607    2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064609    2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064617    2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064620    2557 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064623    2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064625    2557 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064628    2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064630    2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064633    2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064635    2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064638    2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:47:58.072212 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064640    2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064643    2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064645    2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064648    2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064650    2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064653    2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064655    2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064658    2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:58.064660    2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.064666    2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.064792    2557 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.067356    2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.068561    2557 server.go:1019] "Starting client certificate rotation"
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.068659    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:47:58.072813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.069299    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:47:58.073166 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.071775    2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:47:58.073540 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.073522    2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:47:58.083903 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.083878    2557 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:47:58.087613 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.087597    2557 log.go:25] "Validated CRI v1 image API"
Jul 31 14:47:58.088503 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.088489    2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:47:58.092081 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.092055    2557 fs.go:135] Filesystem UUIDs: map[208f0754-9e22-481f-b057-27091a9e1936:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c2b966c1-513e-41e2-bf42-e65ebef70fb9:/dev/nvme0n1p3]
Jul 31 14:47:58.092130 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.092081    2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:47:58.097061 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.096943    2557 manager.go:216] Machine: {Timestamp:2026-07-31 14:47:58.095785165 +0000 UTC m=+0.254036646 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099811 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23431625d2df5a449eb35720a19fe7 SystemUUID:ec234316-25d2-df5a-449e-b35720a19fe7 BootID:c9c2fefc-fb86-4ad5-8747-595195de19d9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ed:16:50:bc:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ed:16:50:bc:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:1c:70:00:97:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:47:58.097061 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.097054    2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:47:58.097174 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.097144    2557 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:47:58.099540 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.099505    2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:47:58.099694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.099544    2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:47:58.100677 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.100665    2557 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:47:58.100713 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.100680    2557 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:47:58.100713 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.100689    2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:47:58.101207 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.101195    2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:47:58.101347 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.101333    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:47:58.101962 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.101950    2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:47:58.106189 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106175    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:47:58.106334 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106323    2557 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:47:58.106375 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106354    2557 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:47:58.106403 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106375    2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:47:58.106403 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106392    2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:47:58.106456 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106406    2557 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:47:58.106456 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.106419    2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:47:58.107425 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.107407    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:47:58.107499 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.107435    2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:47:58.109450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.109431    2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:47:58.110826 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.110814    2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:47:58.110872 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.110844    2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:47:58.111878 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111865    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111885    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111891    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111897    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111905    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111911    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:47:58.111921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111919    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:47:58.112073 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111924    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:47:58.112073 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111931    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:47:58.112073 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111936    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:47:58.112073 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111953    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:47:58.112073 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.111958    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:47:58.112489 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.112480    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:47:58.112489 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.112490    2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:47:58.116868 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.116849    2557 server.go:1268] "Started kubelet"
Jul 31 14:47:58.117021 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.116957    2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:47:58.117091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.116953    2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:47:58.117091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.117046    2557 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:47:58.117382 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.117359    2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:47:58.117873 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.117847    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:47:58.117873 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.117850    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:47:58.117972 ip-10-0-137-96 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:47:58.118334 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.118313    2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:47:58.119121 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.119099    2557 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:47:58.122644 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.122624    2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:47:58.123568 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.123553    2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:47:58.123634 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.123570    2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:47:58.123695 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.123658    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.124099 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124048    2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:47:58.124099 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124073    2557 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:47:58.124099 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124104    2557 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:47:58.124291 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124194    2557 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:47:58.124291 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124205    2557 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:47:58.124475 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124458    2557 factory.go:55] Registering systemd factory
Jul 31 14:47:58.124567 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124481    2557 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:47:58.124754 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124738    2557 factory.go:153] Registering CRI-O factory
Jul 31 14:47:58.124754 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124755    2557 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:47:58.124907 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124804    2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:47:58.124907 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124829    2557 factory.go:103] Registering Raw factory
Jul 31 14:47:58.124907 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.124843    2557 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:47:58.125198 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.124291    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-96.ec2.internal.18c7673a7951136b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-96.ec2.internal,UID:ip-10-0-137-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-96.ec2.internal,},FirstTimestamp:2026-07-31 14:47:58.116811627 +0000 UTC m=+0.275063108,LastTimestamp:2026-07-31 14:47:58.116811627 +0000 UTC m=+0.275063108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-96.ec2.internal,}"
Jul 31 14:47:58.126042 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.125695    2557 manager.go:349] Starting recovery of all containers
Jul 31 14:47:58.128020 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.127986    2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:47:58.128111 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.128078    2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:47:58.135965 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.135835    2557 manager.go:354] Recovery completed
Jul 31 14:47:58.139683 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.139658    2557 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 31 14:47:58.143114 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.143101    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.145510 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.145485    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.145603 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.145529    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.145603 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.145547    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.146133 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146120    2557 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:47:58.146133 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146133    2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:47:58.146222 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146148    2557 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:47:58.146222 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146186    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.146968 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146952    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.147031 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146980    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.147031 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.146991    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.147709 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.147636    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-96.ec2.internal.18c7673a7b06fdcb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-96.ec2.internal,UID:ip-10-0-137-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-96.ec2.internal,},FirstTimestamp:2026-07-31 14:47:58.145510859 +0000 UTC m=+0.303762350,LastTimestamp:2026-07-31 14:47:58.145510859 +0000 UTC m=+0.303762350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-96.ec2.internal,}"
Jul 31 14:47:58.148519 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.148506    2557 policy_none.go:49] "None policy: Start"
Jul 31 14:47:58.148564 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.148523    2557 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:47:58.148564 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.148534    2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:47:58.150443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.150429    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwfgl"
Jul 31 14:47:58.150678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.150669    2557 policy_none.go:47] "Start"
Jul 31 14:47:58.158457 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.158367    2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-96.ec2.internal.18c7673a7b07605e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-96.ec2.internal,UID:ip-10-0-137-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-96.ec2.internal,},FirstTimestamp:2026-07-31 14:47:58.145536094 +0000 UTC m=+0.303787579,LastTimestamp:2026-07-31 14:47:58.145536094 +0000 UTC m=+0.303787579,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-96.ec2.internal,}"
Jul 31 14:47:58.160105 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.160085    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jwfgl"
Jul 31 14:47:58.190969 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.190951    2557 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:47:58.191090 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.190986    2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:47:58.191090 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.190999    2557 server.go:87] "Starting device plugin registration server"
Jul 31 14:47:58.191280 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.191264    2557 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:47:58.191356 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.191284    2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:47:58.191435 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.191405    2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:47:58.191519 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.191500    2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:47:58.191519 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.191514    2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:47:58.192107 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.192089    2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:47:58.192173 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.192137    2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.240006 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.239967    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:47:58.250288 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.241217    2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:47:58.250288 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.241269    2557 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:47:58.250288 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.241298    2557 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:47:58.250288 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.241365    2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:47:58.250288 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.243727    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:58.291813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.291781    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.293157 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.293125    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.293322 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.293161    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.293322 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.293179    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.293322 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.293215    2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.304784 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.304732    2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.304784 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.304763    2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-96.ec2.internal\": node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.337709 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.337677    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.342079 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.342058    2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"]
Jul 31 14:47:58.342140 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.342127    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.343661 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.343639    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.343766 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.343682    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.343766 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.343696    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.346282 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.346269    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.346385 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.346371    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.346429 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.346408    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.347075 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347053    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.347152 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347083    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.347152 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347094    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.347152 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347054    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.347285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347176    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.347285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.347189    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.349630 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.349615    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.349707 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.349640    2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:47:58.350326 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.350296    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:47:58.350406 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.350343    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:47:58.350406 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.350357    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:47:58.374383 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.374356    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-96.ec2.internal\" not found" node="ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.379220 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.379197    2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-96.ec2.internal\" not found" node="ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.437900 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.437860    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.525439 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.525397    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.525439 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.525438    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.525664 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.525465    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf43c7428d90fe73c7fe683bbce5474e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-96.ec2.internal\" (UID: \"cf43c7428d90fe73c7fe683bbce5474e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.538387 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.538340    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.626390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626321    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.626390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626356    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.626390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626381    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf43c7428d90fe73c7fe683bbce5474e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-96.ec2.internal\" (UID: \"cf43c7428d90fe73c7fe683bbce5474e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.626530 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626447    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.626530 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626512    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf43c7428d90fe73c7fe683bbce5474e-config\") pod \"kube-apiserver-proxy-ip-10-0-137-96.ec2.internal\" (UID: \"cf43c7428d90fe73c7fe683bbce5474e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.626599 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.626549    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f3bf0dab2ae39d8d9d30311c040ca5b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal\" (UID: \"6f3bf0dab2ae39d8d9d30311c040ca5b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.639441 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.639387    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.679624 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.679594    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.681916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.681871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:58.739619 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.739573    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.840619 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.840574    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.941233 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:58.941190    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:58.964566 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:58.964535    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:59.041389 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:59.041347    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:59.069663 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.069623    2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:47:59.070223 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.069814    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:47:59.070223 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.069824    2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:47:59.130509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.130474    2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:59.141594 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:47:59.141567    2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-96.ec2.internal\" not found"
Jul 31 14:47:59.164968 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.164910    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:42:58 +0000 UTC" deadline="2028-05-01 06:32:10.217673922 +0000 UTC"
Jul 31 14:47:59.164968 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.164964    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15351h44m11.052712764s"
Jul 31 14:47:59.176653 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.176627    2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:47:59.224540 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.224467    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:59.244125 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.243953    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:47:59.246067 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.246047    2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal"
Jul 31 14:47:59.261336 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.261313    2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:47:59.451580 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:59.451546    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3bf0dab2ae39d8d9d30311c040ca5b.slice/crio-ec30a55e34a560065c001dfc71960be667b98f86febf3ce96aa99776e5cb3e52 WatchSource:0}: Error finding container ec30a55e34a560065c001dfc71960be667b98f86febf3ce96aa99776e5cb3e52: Status 404 returned error can't find the container with id ec30a55e34a560065c001dfc71960be667b98f86febf3ce96aa99776e5cb3e52
Jul 31 14:47:59.453414 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:47:59.453389    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf43c7428d90fe73c7fe683bbce5474e.slice/crio-e4dde2710897ab82bb2808efffbe76b436f3829aebf342fe7d699db40578e9e4 WatchSource:0}: Error finding container e4dde2710897ab82bb2808efffbe76b436f3829aebf342fe7d699db40578e9e4: Status 404 returned error can't find the container with id e4dde2710897ab82bb2808efffbe76b436f3829aebf342fe7d699db40578e9e4
Jul 31 14:47:59.456858 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:47:59.456838    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:48:00.092967 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.092934    2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:00.108456 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.108417    2557 apiserver.go:52] "Watching apiserver"
Jul 31 14:48:00.115543 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.115505    2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:48:00.116927 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.116893    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-zbjsf","openshift-ovn-kubernetes/ovnkube-node-bgwc7","kube-system/global-pull-secret-syncer-jhwrc","kube-system/konnectivity-agent-wmscn","kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal","openshift-multus/multus-additional-cni-plugins-znz82","openshift-network-operator/iptables-alerter-2cctz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966","openshift-cluster-node-tuning-operator/tuned-h528x","openshift-dns/node-resolver-g8ptv","openshift-image-registry/node-ca-2k6ch","openshift-multus/multus-2m6nz","openshift-multus/network-metrics-daemon-6jf5r"]
Jul 31 14:48:00.120508 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.119975    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:00.120508 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.120117    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:00.123294 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.122330    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.123809 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.123788    2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:00.125140 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.125062    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.125248 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.125130    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:00.126590 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.126383    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xg67z\""
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.127447    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.128052    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.128367    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.128532    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.128785    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.128982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.128828    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.129830 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.129612    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:48:00.130053 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.130029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.132541    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133528    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133681    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-netns\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133707    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-ovn\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133726    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-netd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133751    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-dbus\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-var-lib-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.133781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133802    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-systemd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-log-socket\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133846    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-env-overrides\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133881    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovn-node-metrics-cert\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133910    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-script-lib\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133929    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-kubelet-config\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-etc-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-bin\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.133994    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-config\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9l22\" (UniqueName: \"kubernetes.io/projected/3aea51d1-4925-4c7a-993e-a1f8acff1e49-kube-api-access-f9l22\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134041    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-node-log\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:00.134230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134111    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-kubelet\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.135473 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-systemd-units\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.135473 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.134149    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-slash\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.135732    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136539    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136564    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqhlp\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136696    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136788    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136923    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cqlt8\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.136958    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:48:00.137085 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.137006    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rrpbl\""
Jul 31 14:48:00.137644 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.137224    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.137644 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.137501    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:48:00.137811 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.137787    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.138688 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.138668    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:48:00.138889 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.138873    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:48:00.140254 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.140235    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.140772 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.140604    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:48:00.140772 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.140637    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.140989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.140966    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.142994 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.142966    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.143188 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.143168    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-29w9c\""
Jul 31 14:48:00.145590 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.145218    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-49p62\""
Jul 31 14:48:00.145590 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.145269    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.145769 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.145649    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.147333 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.146794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.147333 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.147241    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.147497 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.147386    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tk9qt\""
Jul 31 14:48:00.148409 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.148390    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.148970 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.148621    2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:00.152306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.151773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.152306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.151951    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:48:00.152306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.151965    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qjsqz\""
Jul 31 14:48:00.152306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.152223    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:48:00.152306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.152264    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:48:00.154639 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.154585    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.154738 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.154656    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:00.156093 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.155991    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:48:00.158874 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.158850    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6kr7w\""
Jul 31 14:48:00.165484 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.165459    2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2rdc"
Jul 31 14:48:00.175557 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.175527    2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q2rdc"
Jul 31 14:48:00.225617 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.225589    2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:48:00.234827 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.234796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.234965 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.234848    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-node-log\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.234965 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.234895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:00.234965 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.234932    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-node-log\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.234991    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-systemd-units\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235025    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-slash\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235045    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-systemd-units\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37453-42ef-47fc-97fe-ad3c86bee43f-tmp-dir\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-os-release\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-slash\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235108    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-binary-copy\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.235148 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-netd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235160    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b43d8aa-a300-479e-abab-9ef368b0df29-host\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235185    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-netns\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-netd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jxg\" (UniqueName: \"kubernetes.io/projected/b1b80295-cf3a-4204-9e2f-2703eef3aa50-kube-api-access-j4jxg\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235239    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-tuned\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235307    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-var-lib-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235340    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-script-lib\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtbg\" (UniqueName: \"kubernetes.io/projected/88b37453-42ef-47fc-97fe-ad3c86bee43f-kube-api-access-bmtbg\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235392    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-var-lib-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-etc-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.235572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9l22\" (UniqueName: \"kubernetes.io/projected/3aea51d1-4925-4c7a-993e-a1f8acff1e49-kube-api-access-f9l22\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235583    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-multus-daemon-config\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235612    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxk5\" (UniqueName: \"kubernetes.io/projected/2790d032-2163-47fc-8d82-18241f606df8-kube-api-access-lxxk5\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235641    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-sys-fs\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235666    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-tmp\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-dbus\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235731    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-device-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-kubernetes\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-netns\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235811    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235836    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85t2m\" (UniqueName: \"kubernetes.io/projected/d381633c-bff4-47df-9737-3da37b66a28c-kube-api-access-85t2m\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235862    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysconfig\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235886    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-systemd\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235909    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-lib-modules\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235938    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235967    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-log-socket\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.235993    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovn-node-metrics-cert\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.236165 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-socket-dir-parent\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-script-lib\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236046    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-conf\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236075    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-bin\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236133    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-netns\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236154    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-k8s-cni-cncf-io\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-socket-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-etc-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-cnibin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236251    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cnibin\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236278    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-sys\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-ovn\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236373    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-system-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236398    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34476e65-7f26-44fc-824f-0824aae840c6-konnectivity-ca\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236421    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-registration-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.237071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236490    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-env-overrides\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236507    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236517    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-kubelet\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-dbus\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236549    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brh9\" (UniqueName: \"kubernetes.io/projected/11c53af0-50d3-4ac8-91b6-f4ae52763062-kube-api-access-6brh9\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236586    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-openvswitch\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236640    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-ovn\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.236714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-cni-bin\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.236767    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.236847    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.736822455 +0000 UTC m=+2.895073939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237055    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-env-overrides\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-log-socket\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237068    2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237106    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-modprobe-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.237961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-config\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237267    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b43d8aa-a300-479e-abab-9ef368b0df29-serviceca\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-bin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237473    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-kubelet\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237498    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-multus\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-hostroot\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-host-kubelet\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237624    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-system-cni-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237648    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-host\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237675    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-cni-binary-copy\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237698    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-etc-kubernetes\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237725    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237748    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twbs\" (UniqueName: \"kubernetes.io/projected/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-kube-api-access-8twbs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237771    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44c9\" (UniqueName: \"kubernetes.io/projected/7c766add-e07a-497c-9786-d0a7e5cc397c-kube-api-access-g44c9\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-systemd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.238958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237857    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-kubelet-config\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237890    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-conf-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11c53af0-50d3-4ac8-91b6-f4ae52763062-host-slash\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237945    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3aea51d1-4925-4c7a-993e-a1f8acff1e49-run-systemd\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237948    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-run\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/368ae851-9ae7-4d58-8715-aa95f856f689-kubelet-config\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.237986    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-var-lib-kubelet\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238109    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88b37453-42ef-47fc-97fe-ad3c86bee43f-hosts-file\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/3b43d8aa-a300-479e-abab-9ef368b0df29-kube-api-access-dcvhk\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238163    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-os-release\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-multus-certs\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34476e65-7f26-44fc-824f-0824aae840c6-agent-certs\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238310    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11c53af0-50d3-4ac8-91b6-f4ae52763062-iptables-alerter-script\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.239832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.238417    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovnkube-config\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.240779 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.240732    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3aea51d1-4925-4c7a-993e-a1f8acff1e49-ovn-node-metrics-cert\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.244455 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.243781    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:00.244455 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.243808    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:00.244455 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.243822    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.244455 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.243886    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.743866473 +0000 UTC m=+2.902117966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.246393 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.246373    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9l22\" (UniqueName: \"kubernetes.io/projected/3aea51d1-4925-4c7a-993e-a1f8acff1e49-kube-api-access-f9l22\") pod \"ovnkube-node-bgwc7\" (UID: \"3aea51d1-4925-4c7a-993e-a1f8acff1e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.247221 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.247165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal" event={"ID":"cf43c7428d90fe73c7fe683bbce5474e","Type":"ContainerStarted","Data":"e4dde2710897ab82bb2808efffbe76b436f3829aebf342fe7d699db40578e9e4"}
Jul 31 14:48:00.249673 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.249596    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal" event={"ID":"6f3bf0dab2ae39d8d9d30311c040ca5b","Type":"ContainerStarted","Data":"ec30a55e34a560065c001dfc71960be667b98f86febf3ce96aa99776e5cb3e52"}
Jul 31 14:48:00.338961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.338910    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-system-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.338977    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34476e65-7f26-44fc-824f-0824aae840c6-konnectivity-ca\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-registration-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339039    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-kubelet\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339070    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brh9\" (UniqueName: \"kubernetes.io/projected/11c53af0-50d3-4ac8-91b6-f4ae52763062-kube-api-access-6brh9\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339125    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-modprobe-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.339167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339153    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b43d8aa-a300-479e-abab-9ef368b0df29-serviceca\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-bin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339230    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-multus\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339255    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-hostroot\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339299    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-system-cni-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339342    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-host\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339376    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-cni-binary-copy\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339402    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-etc-kubernetes\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339435    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twbs\" (UniqueName: \"kubernetes.io/projected/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-kube-api-access-8twbs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44c9\" (UniqueName: \"kubernetes.io/projected/7c766add-e07a-497c-9786-d0a7e5cc397c-kube-api-access-g44c9\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339518    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-conf-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339572    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11c53af0-50d3-4ac8-91b6-f4ae52763062-host-slash\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339598    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-run\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339626    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-var-lib-kubelet\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.339654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339654    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88b37453-42ef-47fc-97fe-ad3c86bee43f-hosts-file\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339680    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/3b43d8aa-a300-479e-abab-9ef368b0df29-kube-api-access-dcvhk\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-os-release\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-multus-certs\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339757    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34476e65-7f26-44fc-824f-0824aae840c6-agent-certs\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339756    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-hostroot\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339788    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11c53af0-50d3-4ac8-91b6-f4ae52763062-iptables-alerter-script\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339838    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-system-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339911    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37453-42ef-47fc-97fe-ad3c86bee43f-tmp-dir\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339940    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-os-release\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11c53af0-50d3-4ac8-91b6-f4ae52763062-host-slash\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339968    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-binary-copy\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.339999    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b43d8aa-a300-479e-abab-9ef368b0df29-host\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-system-cni-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-netns\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.340491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340066    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-host\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340074    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jxg\" (UniqueName: \"kubernetes.io/projected/b1b80295-cf3a-4204-9e2f-2703eef3aa50-kube-api-access-j4jxg\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-run\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340132    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-tuned\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340162    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtbg\" (UniqueName: \"kubernetes.io/projected/88b37453-42ef-47fc-97fe-ad3c86bee43f-kube-api-access-bmtbg\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340172    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-var-lib-kubelet\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340191    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340220    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-multus-daemon-config\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340244    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxk5\" (UniqueName: \"kubernetes.io/projected/2790d032-2163-47fc-8d82-18241f606df8-kube-api-access-lxxk5\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340256    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340271    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-sys-fs\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340296    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-tmp\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340335    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-device-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340360    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-kubernetes\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340380    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88b37453-42ef-47fc-97fe-ad3c86bee43f-hosts-file\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340387    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340434    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85t2m\" (UniqueName: \"kubernetes.io/projected/d381633c-bff4-47df-9737-3da37b66a28c-kube-api-access-85t2m\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.341285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340459    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysconfig\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340483    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-systemd\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340509    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-lib-modules\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340542    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-cni-binary-copy\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340565    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-socket-dir-parent\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-etc-kubernetes\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340593    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-conf\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340623    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-lib-modules\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.340655    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.340697    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:00.84068434 +0000 UTC m=+2.998935810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340706    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-multus\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340714    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysctl-conf\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340723    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-os-release\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340767    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-kubelet\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340808    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/34476e65-7f26-44fc-824f-0824aae840c6-konnectivity-ca\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340871    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-conf-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.342197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340895    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-k8s-cni-cncf-io\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340914    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-socket-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340934    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-cnibin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cnibin\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340966    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.340982    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-sys\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341043    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-sys\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341070    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-multus-certs\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88b37453-42ef-47fc-97fe-ad3c86bee43f-tmp-dir\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341108    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-os-release\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341134    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-k8s-cni-cncf-io\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341147    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-modprobe-d\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-socket-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341240    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-cnibin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341264    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cnibin\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341339    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-binary-copy\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11c53af0-50d3-4ac8-91b6-f4ae52763062-iptables-alerter-script\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341391    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-socket-dir-parent\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.343173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341397    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1b80295-cf3a-4204-9e2f-2703eef3aa50-tuning-conf-dir\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341419    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b43d8aa-a300-479e-abab-9ef368b0df29-host\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-sys-fs\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341499    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-run-netns\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341635    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b43d8aa-a300-479e-abab-9ef368b0df29-serviceca\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341707    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-host-var-lib-cni-bin\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.341894    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342209    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2790d032-2163-47fc-8d82-18241f606df8-multus-daemon-config\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342277    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-systemd\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342295    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-device-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342341    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-sysconfig\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342377    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-kubernetes\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342401    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2790d032-2163-47fc-8d82-18241f606df8-multus-cni-dir\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342437    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d381633c-bff4-47df-9737-3da37b66a28c-registration-dir\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.344642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.342525    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1b80295-cf3a-4204-9e2f-2703eef3aa50-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.345847 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.345817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/34476e65-7f26-44fc-824f-0824aae840c6-agent-certs\") pod \"konnectivity-agent-wmscn\" (UID: \"34476e65-7f26-44fc-824f-0824aae840c6\") " pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.346103 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.346080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-tmp\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.347999 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.347939    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c766add-e07a-497c-9786-d0a7e5cc397c-etc-tuned\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.363096 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.363065    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvhk\" (UniqueName: \"kubernetes.io/projected/3b43d8aa-a300-479e-abab-9ef368b0df29-kube-api-access-dcvhk\") pod \"node-ca-2k6ch\" (UID: \"3b43d8aa-a300-479e-abab-9ef368b0df29\") " pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.366058 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.366029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44c9\" (UniqueName: \"kubernetes.io/projected/7c766add-e07a-497c-9786-d0a7e5cc397c-kube-api-access-g44c9\") pod \"tuned-h528x\" (UID: \"7c766add-e07a-497c-9786-d0a7e5cc397c\") " pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.370006 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.369974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85t2m\" (UniqueName: \"kubernetes.io/projected/d381633c-bff4-47df-9737-3da37b66a28c-kube-api-access-85t2m\") pod \"aws-ebs-csi-driver-node-tl966\" (UID: \"d381633c-bff4-47df-9737-3da37b66a28c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.370375 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.370312    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxk5\" (UniqueName: \"kubernetes.io/projected/2790d032-2163-47fc-8d82-18241f606df8-kube-api-access-lxxk5\") pod \"multus-2m6nz\" (UID: \"2790d032-2163-47fc-8d82-18241f606df8\") " pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.372024 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.371078    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtbg\" (UniqueName: \"kubernetes.io/projected/88b37453-42ef-47fc-97fe-ad3c86bee43f-kube-api-access-bmtbg\") pod \"node-resolver-g8ptv\" (UID: \"88b37453-42ef-47fc-97fe-ad3c86bee43f\") " pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.372024 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.372000    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brh9\" (UniqueName: \"kubernetes.io/projected/11c53af0-50d3-4ac8-91b6-f4ae52763062-kube-api-access-6brh9\") pod \"iptables-alerter-2cctz\" (UID: \"11c53af0-50d3-4ac8-91b6-f4ae52763062\") " pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.372482 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.372463    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jxg\" (UniqueName: \"kubernetes.io/projected/b1b80295-cf3a-4204-9e2f-2703eef3aa50-kube-api-access-j4jxg\") pod \"multus-additional-cni-plugins-znz82\" (UID: \"b1b80295-cf3a-4204-9e2f-2703eef3aa50\") " pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.375077 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.374549    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twbs\" (UniqueName: \"kubernetes.io/projected/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-kube-api-access-8twbs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.443888 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.443850    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:00.452642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.452616    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:00.466086 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.466055    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-znz82"
Jul 31 14:48:00.472467 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.472356    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2cctz"
Jul 31 14:48:00.482395 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.482354    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966"
Jul 31 14:48:00.490728 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.490526    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h528x"
Jul 31 14:48:00.500759 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.500733    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g8ptv"
Jul 31 14:48:00.511010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.510980    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2k6ch"
Jul 31 14:48:00.519120 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.519085    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2m6nz"
Jul 31 14:48:00.744794 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.744749    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:00.744985 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.744817    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:00.744985 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.744931    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.744985 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.744942    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:00.744985 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.744961    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:00.744985 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.744975    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.745240 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.745004    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:01.744985152 +0000 UTC m=+3.903236648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:00.745240 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.745023    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:01.745016059 +0000 UTC m=+3.903267530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:00.845774 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:00.845733    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:00.845958 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.845860    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:00.845958 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:00.845929    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:01.845908664 +0000 UTC m=+4.004160136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:01.157751 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.157665    2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:01.179376 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.179330    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:00 +0000 UTC" deadline="2028-03-24 09:47:07.265488211 +0000 UTC"
Jul 31 14:48:01.179376 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.179372    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14442h59m6.086120582s"
Jul 31 14:48:01.242602 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.242562    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:01.242781 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.242683    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:01.504497 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.504342    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c766add_e07a_497c_9786_d0a7e5cc397c.slice/crio-0ae12df4c06770561d4d27b339c76ff56857dcef54b971f32f8b15f3480e58b5 WatchSource:0}: Error finding container 0ae12df4c06770561d4d27b339c76ff56857dcef54b971f32f8b15f3480e58b5: Status 404 returned error can't find the container with id 0ae12df4c06770561d4d27b339c76ff56857dcef54b971f32f8b15f3480e58b5
Jul 31 14:48:01.507976 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.507945    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b37453_42ef_47fc_97fe_ad3c86bee43f.slice/crio-fc6ef8e2739367ba5b9138f146908c9b1dec66d45e5ecda75eefd2eae93abbb9 WatchSource:0}: Error finding container fc6ef8e2739367ba5b9138f146908c9b1dec66d45e5ecda75eefd2eae93abbb9: Status 404 returned error can't find the container with id fc6ef8e2739367ba5b9138f146908c9b1dec66d45e5ecda75eefd2eae93abbb9
Jul 31 14:48:01.511833 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.511807    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34476e65_7f26_44fc_824f_0824aae840c6.slice/crio-d14da043bd4ef853f4a627236b50599daa92443ed8851344eabf0ac97943eb8b WatchSource:0}: Error finding container d14da043bd4ef853f4a627236b50599daa92443ed8851344eabf0ac97943eb8b: Status 404 returned error can't find the container with id d14da043bd4ef853f4a627236b50599daa92443ed8851344eabf0ac97943eb8b
Jul 31 14:48:01.512071 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.512044    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b43d8aa_a300_479e_abab_9ef368b0df29.slice/crio-fa4c91bd3d19e1251fc2a127e43edbe0a3ed802d1bb8d6036d48724d5378303a WatchSource:0}: Error finding container fa4c91bd3d19e1251fc2a127e43edbe0a3ed802d1bb8d6036d48724d5378303a: Status 404 returned error can't find the container with id fa4c91bd3d19e1251fc2a127e43edbe0a3ed802d1bb8d6036d48724d5378303a
Jul 31 14:48:01.513785 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.513756    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c53af0_50d3_4ac8_91b6_f4ae52763062.slice/crio-4457826ba14c55d1b71b5369d5ef26270dbcadcde78747eb381e1cd181c750ce WatchSource:0}: Error finding container 4457826ba14c55d1b71b5369d5ef26270dbcadcde78747eb381e1cd181c750ce: Status 404 returned error can't find the container with id 4457826ba14c55d1b71b5369d5ef26270dbcadcde78747eb381e1cd181c750ce
Jul 31 14:48:01.514951 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.514635    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd381633c_bff4_47df_9737_3da37b66a28c.slice/crio-3d90e7cff65647af0f458864f18e8148a5ea20d62e0f929ef44471eb147d7691 WatchSource:0}: Error finding container 3d90e7cff65647af0f458864f18e8148a5ea20d62e0f929ef44471eb147d7691: Status 404 returned error can't find the container with id 3d90e7cff65647af0f458864f18e8148a5ea20d62e0f929ef44471eb147d7691
Jul 31 14:48:01.516874 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:01.516135    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2790d032_2163_47fc_8d82_18241f606df8.slice/crio-f6601d94d14ac1d3bc920a609092efc803f8eb240373ff9ae9469c2807ad3bc5 WatchSource:0}: Error finding container f6601d94d14ac1d3bc920a609092efc803f8eb240373ff9ae9469c2807ad3bc5: Status 404 returned error can't find the container with id f6601d94d14ac1d3bc920a609092efc803f8eb240373ff9ae9469c2807ad3bc5
Jul 31 14:48:01.754734 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.754560    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:01.754856 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754730    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:01.754856 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754755    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:01.754856 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754766    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:01.754856 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.754771    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:01.754856 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754820    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:03.754805746 +0000 UTC m=+5.913057221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:01.755051 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754860    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:01.755051 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.754911    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:03.754900281 +0000 UTC m=+5.913151750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:01.855543 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:01.855512    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:01.855695 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.855674    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:01.855759 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:01.855749    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:03.855732985 +0000 UTC m=+6.013984455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:02.179648 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.179575    2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:00 +0000 UTC" deadline="2028-04-09 04:02:12.92983544 +0000 UTC"
Jul 31 14:48:02.179648 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.179646    2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14821h14m10.750193592s"
Jul 31 14:48:02.242473 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.242444    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:02.242670 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:02.242597    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:02.242670 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.242642    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:02.242798 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:02.242764    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:02.256549 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.256422    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wmscn" event={"ID":"34476e65-7f26-44fc-824f-0824aae840c6","Type":"ContainerStarted","Data":"d14da043bd4ef853f4a627236b50599daa92443ed8851344eabf0ac97943eb8b"}
Jul 31 14:48:02.260578 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.260539    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal" event={"ID":"cf43c7428d90fe73c7fe683bbce5474e","Type":"ContainerStarted","Data":"2fc4702ea504167a082a10a3923436c807a675d354039ec8164d5a3e98b8a78d"}
Jul 31 14:48:02.265781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.265748    2557 generic.go:356] "Generic (PLEG): container finished" podID="6f3bf0dab2ae39d8d9d30311c040ca5b" containerID="89a439acd33931376b8e6aae4f48cdd785a2f64ac32793ba7ec88df70be59a86" exitCode=0
Jul 31 14:48:02.265937 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.265839    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal" event={"ID":"6f3bf0dab2ae39d8d9d30311c040ca5b","Type":"ContainerDied","Data":"89a439acd33931376b8e6aae4f48cdd785a2f64ac32793ba7ec88df70be59a86"}
Jul 31 14:48:02.269163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.269130    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerStarted","Data":"15c55b2a9aa19183ec6c68ea11d46a6c906ad946c7fab6d220adec118ab58545"}
Jul 31 14:48:02.271524 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.271495    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2m6nz" event={"ID":"2790d032-2163-47fc-8d82-18241f606df8","Type":"ContainerStarted","Data":"f6601d94d14ac1d3bc920a609092efc803f8eb240373ff9ae9469c2807ad3bc5"}
Jul 31 14:48:02.274479 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.274452    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"10f3edb57751901cecff08da5ed950e31bdc341d54e616d9b2991ae75761b3dd"}
Jul 31 14:48:02.295666 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.295630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2cctz" event={"ID":"11c53af0-50d3-4ac8-91b6-f4ae52763062","Type":"ContainerStarted","Data":"4457826ba14c55d1b71b5369d5ef26270dbcadcde78747eb381e1cd181c750ce"}
Jul 31 14:48:02.300116 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.300085    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8ptv" event={"ID":"88b37453-42ef-47fc-97fe-ad3c86bee43f","Type":"ContainerStarted","Data":"fc6ef8e2739367ba5b9138f146908c9b1dec66d45e5ecda75eefd2eae93abbb9"}
Jul 31 14:48:02.300298 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.300241    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-96.ec2.internal" podStartSLOduration=3.300211293 podStartE2EDuration="3.300211293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:02.279599573 +0000 UTC m=+4.437851067" watchObservedRunningTime="2026-07-31 14:48:02.300211293 +0000 UTC m=+4.458462788"
Jul 31 14:48:02.305364 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.305333    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h528x" event={"ID":"7c766add-e07a-497c-9786-d0a7e5cc397c","Type":"ContainerStarted","Data":"0ae12df4c06770561d4d27b339c76ff56857dcef54b971f32f8b15f3480e58b5"}
Jul 31 14:48:02.309229 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.309198    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966" event={"ID":"d381633c-bff4-47df-9737-3da37b66a28c","Type":"ContainerStarted","Data":"3d90e7cff65647af0f458864f18e8148a5ea20d62e0f929ef44471eb147d7691"}
Jul 31 14:48:02.312624 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:02.312582    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2k6ch" event={"ID":"3b43d8aa-a300-479e-abab-9ef368b0df29","Type":"ContainerStarted","Data":"fa4c91bd3d19e1251fc2a127e43edbe0a3ed802d1bb8d6036d48724d5378303a"}
Jul 31 14:48:03.241562 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.241522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:03.242111 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.241659    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:03.348384 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.348235    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal" event={"ID":"6f3bf0dab2ae39d8d9d30311c040ca5b","Type":"ContainerStarted","Data":"2187a3d3967d93cce287c6c75abd620cd2485e53981e53628dba806b170fa3f2"}
Jul 31 14:48:03.373456 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.373394    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-96.ec2.internal" podStartSLOduration=4.373371787 podStartE2EDuration="4.373371787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:03.372623975 +0000 UTC m=+5.530875469" watchObservedRunningTime="2026-07-31 14:48:03.373371787 +0000 UTC m=+5.531623281"
Jul 31 14:48:03.770151 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.769699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:03.770151 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.769770    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:03.770151 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.769900    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:03.770151 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.769966    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:07.769947045 +0000 UTC m=+9.928198523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:03.770629 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.770491    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:03.770629 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.770514    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:03.770629 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.770528    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:03.770629 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.770579    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:07.770561657 +0000 UTC m=+9.928813133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:03.871010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:03.870874    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:03.871212 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.871051    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:03.871212 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:03.871135    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:07.871110621 +0000 UTC m=+10.029362114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:04.244460 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:04.244429    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:04.244936 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:04.244578    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:04.245197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:04.244299    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:04.245350 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:04.245327    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:05.242713 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:05.242672    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:05.242892 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:05.242827    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:06.241930 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:06.241897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:06.242478 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:06.241897    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:06.242478 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:06.242061    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:06.242478 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:06.242132    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:07.242528 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:07.242491    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:07.243039 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.242639    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:07.802743 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:07.802705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:07.802937 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:07.802769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:07.802937 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.802926    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:07.803057 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.802990    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:15.802971269 +0000 UTC m=+17.961222740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:07.803492 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.803375    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:07.803492 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.803398    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:07.803492 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.803414    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:07.803492 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.803462    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:15.803447668 +0000 UTC m=+17.961699151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:07.903823 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:07.903789    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:07.903996 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.903978    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:07.904053 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:07.904041    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:15.904021646 +0000 UTC m=+18.062273117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:08.242551 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:08.242518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:08.242993 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:08.242645    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:08.243068 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:08.243045    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:08.243156 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:08.243137    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:09.241903 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:09.241866    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:09.242092 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:09.242006    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:10.246474 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:10.245807    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:10.246474 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:10.245927    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:10.246474 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:10.246303    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:10.246474 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:10.246434    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:11.241958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:11.241923    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:11.242148 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:11.242059    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:12.242030 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:12.241992    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:12.242422 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:12.242111    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:12.242422 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:12.242170    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:12.242422 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:12.242296    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:13.241513 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:13.241477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:13.241685 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:13.241609    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:14.244206 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:14.244153    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:14.244634 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:14.244215    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:14.244634 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:14.244310    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:14.244634 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:14.244447    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:15.242409 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:15.242374    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:15.242627 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.242491    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:15.866672 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:15.866627    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:15.866704    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866804    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866835    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866847    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866865    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866914    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:31.866899893 +0000 UTC m=+34.025151360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:15.867134 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.866937    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:31.866924703 +0000 UTC m=+34.025176177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:15.967702 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:15.967665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:15.967898 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.967877    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:15.967970 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:15.967956    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:31.967936393 +0000 UTC m=+34.126187865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:16.242157 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:16.242123    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:16.242403 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:16.242292    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:16.242403 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:16.242373    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:16.242589 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:16.242518    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:17.242255 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:17.242211    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:17.242707 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:17.242357    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:18.242933 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:18.242884    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:18.243460 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:18.242992    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:18.243460 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:18.243057    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:18.243460 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:18.243171    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:19.241719 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:19.241690    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:19.241925 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:19.241806    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:20.241724 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:20.241682    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:20.242310 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:20.241691    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:20.242310 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:20.241792    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:20.242310 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:20.241878    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:21.241840 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.241518    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:21.242510 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:21.241930    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:21.394163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.394122    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2k6ch" event={"ID":"3b43d8aa-a300-479e-abab-9ef368b0df29","Type":"ContainerStarted","Data":"01831393fcb878befc3a533eceb931d9ef9b4bc1a37b9ce7d4d9b81c43628511"}
Jul 31 14:48:21.395756 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.395726    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wmscn" event={"ID":"34476e65-7f26-44fc-824f-0824aae840c6","Type":"ContainerStarted","Data":"7e31928eced6cc57b9b73483cd5de0dcad177248ec05a175fecb2b99a1ded3ee"}
Jul 31 14:48:21.397479 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.397449    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="afe4984d4626203400b8e83aead90b8808fdcdd53c59702c352f1712b6942fdf" exitCode=0
Jul 31 14:48:21.397574 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.397532    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"afe4984d4626203400b8e83aead90b8808fdcdd53c59702c352f1712b6942fdf"}
Jul 31 14:48:21.399592 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.399564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2m6nz" event={"ID":"2790d032-2163-47fc-8d82-18241f606df8","Type":"ContainerStarted","Data":"ed14324d7ccbd97e8cbbbba795c22ccb17654c5897ac36e99dc57db878d28aae"}
Jul 31 14:48:21.402776 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.402753    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"300a1763e206e8328b648290af306e0c225552398e0f225caba375ccbb42b856"}
Jul 31 14:48:21.402869 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.402824    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"699cfbd25b1d77d96c6d9659b3ded7926f98f6368cc379a0dc838f8101b4bcca"}
Jul 31 14:48:21.402938 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.402874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"25ee05320d73fbe4cea16ed6142a0c4df34fa66376e7751282e6d9d553ef1973"}
Jul 31 14:48:21.402938 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.402923    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"88d8c493ded27f8f3a809b810377871ea827b5b9689298306582d00039f02b73"}
Jul 31 14:48:21.403047 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.402969    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"5b8119ebe5828f4e117d4ff782aef6d603035961de66f9e2422230c02b45815b"}
Jul 31 14:48:21.403047 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.403017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"3ec9b1ea407804a5d0e52c8d838856fb3a67aab77b8e9597ebe6b3b339997322"}
Jul 31 14:48:21.404367 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.404342    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g8ptv" event={"ID":"88b37453-42ef-47fc-97fe-ad3c86bee43f","Type":"ContainerStarted","Data":"2f646c4146abdb8345a9ce9a4630a10c081d34aedb607b432332ece7cd8df738"}
Jul 31 14:48:21.407136 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.407112    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h528x" event={"ID":"7c766add-e07a-497c-9786-d0a7e5cc397c","Type":"ContainerStarted","Data":"557d4c90ea45951693580915615404a028b50dfc542ca6b9f0aa04c3fc2eacbc"}
Jul 31 14:48:21.409419 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.409397    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966" event={"ID":"d381633c-bff4-47df-9737-3da37b66a28c","Type":"ContainerStarted","Data":"a6174f8cc35100ffd19bd1ae53fd6e94c08265198ddf37127fa67445ece22535"}
Jul 31 14:48:21.417429 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.417376    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2k6ch" podStartSLOduration=3.574130099 podStartE2EDuration="22.417362628s" totalImagesPullingTime="18.843232529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.416532525 +0000 UTC m=+23.574784018" watchObservedRunningTime="2026-07-31 14:48:21.417362628 +0000 UTC m=+23.575614118"
Jul 31 14:48:21.462147 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.462093    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2m6nz" podStartSLOduration=3.509368661 podStartE2EDuration="22.462075068s" totalImagesPullingTime="18.952706407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.461970751 +0000 UTC m=+23.620222241" watchObservedRunningTime="2026-07-31 14:48:21.462075068 +0000 UTC m=+23.620326559"
Jul 31 14:48:21.462755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.462724    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g8ptv" podStartSLOduration=3.563624958 podStartE2EDuration="22.462715708s" totalImagesPullingTime="18.89909075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.436039339 +0000 UTC m=+23.594290820" watchObservedRunningTime="2026-07-31 14:48:21.462715708 +0000 UTC m=+23.620967197"
Jul 31 14:48:21.488324 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.488269    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wmscn" podStartSLOduration=3.592394612 podStartE2EDuration="22.488254518s" totalImagesPullingTime="18.895859906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.488051655 +0000 UTC m=+23.646303140" watchObservedRunningTime="2026-07-31 14:48:21.488254518 +0000 UTC m=+23.646506000"
Jul 31 14:48:21.566554 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.566501    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h528x" podStartSLOduration=3.649528423 podStartE2EDuration="22.566482965s" totalImagesPullingTime="18.916954542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:21.565823828 +0000 UTC m=+23.724075319" watchObservedRunningTime="2026-07-31 14:48:21.566482965 +0000 UTC m=+23.724734455"
Jul 31 14:48:21.956314 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:21.956290    2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:22.202477 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.202284    2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:21.956308398Z","UUID":"d3de9b66-4bc5-42d5-a705-9141e26ca4ad","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:22.203585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.203565    2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:22.203706 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.203600    2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:22.241636 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.241607    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:22.241800 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:22.241714    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:22.241847 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.241814    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:22.241959 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:22.241937    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:22.413339 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.413304    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2cctz" event={"ID":"11c53af0-50d3-4ac8-91b6-f4ae52763062","Type":"ContainerStarted","Data":"b67c61d7554f3f23719d7b40e311bac0c1d43512e68612a4f2b3bd4ad8b64d0f"}
Jul 31 14:48:22.415012 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:22.414982    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966" event={"ID":"d381633c-bff4-47df-9737-3da37b66a28c","Type":"ContainerStarted","Data":"4a4668b64519d0df77c55169ba110f3a036e9b8550659c917bbdc65902319d6b"}
Jul 31 14:48:23.242123 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:23.241938    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:23.242536 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:23.242224    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:23.420205 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:23.420168    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966" event={"ID":"d381633c-bff4-47df-9737-3da37b66a28c","Type":"ContainerStarted","Data":"66070bd8817962b6488e60c94d460084b8abafbc9d994fb6b45f383a0e775f02"}
Jul 31 14:48:23.424052 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:23.424028    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"b5c548a47b44d2b567ac2837ec9abc8f25791fabb4354de015d79bf177ff8fd7"}
Jul 31 14:48:23.469076 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:23.469027    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2cctz" podStartSLOduration=5.571656004 podStartE2EDuration="24.469010901s" totalImagesPullingTime="18.897354897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:22.438847356 +0000 UTC m=+24.597098845" watchObservedRunningTime="2026-07-31 14:48:23.469010901 +0000 UTC m=+25.627262393"
Jul 31 14:48:23.469280 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:23.469118    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tl966" podStartSLOduration=3.287747306 podStartE2EDuration="24.469110899s" totalImagesPullingTime="21.181363593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:23.462396657 +0000 UTC m=+25.620648150" watchObservedRunningTime="2026-07-31 14:48:23.469110899 +0000 UTC m=+25.627362389"
Jul 31 14:48:24.242391 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:24.242348    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:24.242887 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:24.242348    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:24.242887 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:24.242500    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:24.242887 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:24.242528    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:25.241902 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:25.241871    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:25.242076 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:25.241992    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:26.200402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.200364    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:26.201067 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.200901    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:26.242475 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.242446    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:26.242648 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:26.242581    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:26.242648 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.242626    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:26.242761 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:26.242714    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:26.437364 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.437172    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" event={"ID":"3aea51d1-4925-4c7a-993e-a1f8acff1e49","Type":"ContainerStarted","Data":"1f7c0694ee1671ddfe207aae386973bb3c353f22a2e729ffcac71587b56d533b"}
Jul 31 14:48:26.437510 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.437447    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:26.438005 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.437985    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wmscn"
Jul 31 14:48:26.484160 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:26.484098    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7" podStartSLOduration=8.526661274 podStartE2EDuration="27.484078513s" totalImagesPullingTime="18.957417239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:26.48367829 +0000 UTC m=+28.641929781" watchObservedRunningTime="2026-07-31 14:48:26.484078513 +0000 UTC m=+28.642330009"
Jul 31 14:48:27.242041 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.242003    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:27.242644 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:27.242115    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:27.440876 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.440836    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="4570f640a67f9e909d1c911f06af2be431e46df74bcf1f610386cdc18ed2f7cf" exitCode=0
Jul 31 14:48:27.441061 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.440920    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"4570f640a67f9e909d1c911f06af2be431e46df74bcf1f610386cdc18ed2f7cf"}
Jul 31 14:48:27.441400 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.441383    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:27.442195 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.441521    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:27.442195 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.441589    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:27.456958 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.456937    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:27.457044 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:27.457028    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:28.242510 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.242478    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:28.242972 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:28.242567    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:28.242972 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.242613    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:28.242972 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:28.242702    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:28.351386 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.351353    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbjsf"]
Jul 31 14:48:28.351525 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.351492    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:28.351616 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:28.351587    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:28.359533 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.359499    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jf5r"]
Jul 31 14:48:28.367561 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.367533    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhwrc"]
Jul 31 14:48:28.445441 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.445419    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:28.445560 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.445477    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerStarted","Data":"3be39bc767f63965a008c5da9fc2e5df44cd77a2ea60ff115cf45089c287df8c"}
Jul 31 14:48:28.445560 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:28.445536    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:28.446205 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:28.446185    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:28.446376 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:28.446351    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:29.449565 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:29.449531    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="3be39bc767f63965a008c5da9fc2e5df44cd77a2ea60ff115cf45089c287df8c" exitCode=0
Jul 31 14:48:29.449931 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:29.449617    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"3be39bc767f63965a008c5da9fc2e5df44cd77a2ea60ff115cf45089c287df8c"}
Jul 31 14:48:30.242488 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:30.242232    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:30.242666 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:30.242318    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:30.242666 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:30.242533    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:30.242666 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:30.242588    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:30.242666 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:30.242233    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:30.242666 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:30.242666    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:30.453688 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:30.453657    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerStarted","Data":"4cbad830d1d8c96599b820f12463ce978eba16cd8b7651b565e81f0e55e043fe"}
Jul 31 14:48:31.457777 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:31.457744    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="4cbad830d1d8c96599b820f12463ce978eba16cd8b7651b565e81f0e55e043fe" exitCode=0
Jul 31 14:48:31.458131 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:31.457801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"4cbad830d1d8c96599b820f12463ce978eba16cd8b7651b565e81f0e55e043fe"}
Jul 31 14:48:31.887565 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:31.887530    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:31.887565 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:31.887571    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:31.887772 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887699    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:31.887772 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887719    2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:31.887772 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887729    2557 projected.go:196] Error preparing data for projected volume kube-api-access-wghjf for pod openshift-network-diagnostics/network-check-target-zbjsf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:31.887772 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887734    2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:31.887889 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887780    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf podName:00464f4b-81cb-4133-8f58-e017c9625482 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.887766979 +0000 UTC m=+66.046018447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wghjf" (UniqueName: "kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf") pod "network-check-target-zbjsf" (UID: "00464f4b-81cb-4133-8f58-e017c9625482") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:31.887889 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.887793    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret podName:368ae851-9ae7-4d58-8715-aa95f856f689 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.88778746 +0000 UTC m=+66.046038929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret") pod "global-pull-secret-syncer-jhwrc" (UID: "368ae851-9ae7-4d58-8715-aa95f856f689") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:31.988809 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:31.988777    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:31.988996 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.988946    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:31.989054 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:31.989008    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:03.988993914 +0000 UTC m=+66.147245405 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:32.242094 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:32.242009    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:32.242094 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:32.242051    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:32.242376 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:32.242132    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:32.242376 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:32.242139    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jhwrc" podUID="368ae851-9ae7-4d58-8715-aa95f856f689"
Jul 31 14:48:32.242376 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:32.242231    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zbjsf" podUID="00464f4b-81cb-4133-8f58-e017c9625482"
Jul 31 14:48:32.242376 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:32.242331    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jf5r" podUID="f1ad65b5-6b4d-4d71-8ca8-b9326a58b567"
Jul 31 14:48:33.252938 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.252855    2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-96.ec2.internal" event="NodeReady"
Jul 31 14:48:33.253452 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.253060    2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:33.300783 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.300748    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:48:33.304553 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.304534    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.307466 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.307442    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 14:48:33.307572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.307544    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 14:48:33.307572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.307555    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 14:48:33.307572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.307567    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plk7v\""
Jul 31 14:48:33.314004 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.313978    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 14:48:33.328684 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.328258    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:48:33.328925 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.328882    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tvb95"]
Jul 31 14:48:33.332852 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.332828    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.336515 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.336491    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:48:33.336642 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.336521    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:48:33.336713 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.336657    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:48:33.337000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.336979    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9kr2\""
Jul 31 14:48:33.347305 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.347277    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tvb95"]
Jul 31 14:48:33.401046 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401011    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401268 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401075    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401268 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401130    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401268 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401171    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401268 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401206    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401495 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401300    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2ch\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401495 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401341    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.401495 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.401356    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.434687 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.434657    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lkpw6"]
Jul 31 14:48:33.438615 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.438588    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.444045 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.443989    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:48:33.444193 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.444162    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p6mpf\""
Jul 31 14:48:33.444272 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.444219    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:48:33.475134 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.475102    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkpw6"]
Jul 31 14:48:33.502364 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502328    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2ch\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502369    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgjj\" (UniqueName: \"kubernetes.io/projected/58d74f63-ce55-4673-9752-584c1d65522f-kube-api-access-xjgjj\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.502534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502418    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502441    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58d74f63-ce55-4673-9752-584c1d65522f-cert\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.502727 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502727 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502727 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502867 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502745    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502867 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502774    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.502867 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:33.502841    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:33.502867 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:33.502864    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:33.503089 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.502898    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.503089 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:33.502931    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.002910247 +0000 UTC m=+36.161161729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:33.503353 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.503334    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.503536 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.503516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.507312 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.507288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.507705 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.507685    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.523068 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.523037    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2ch\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.523302 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.523282    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:33.604190 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604151    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58d74f63-ce55-4673-9752-584c1d65522f-cert\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.604372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604263    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-config-volume\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.604372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604289    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgjj\" (UniqueName: \"kubernetes.io/projected/58d74f63-ce55-4673-9752-584c1d65522f-kube-api-access-xjgjj\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.604372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.604372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604340    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-tmp-dir\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.604585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.604417    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9kw\" (UniqueName: \"kubernetes.io/projected/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-kube-api-access-2p9kw\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.606760 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.606736    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58d74f63-ce55-4673-9752-584c1d65522f-cert\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.621728 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.621702    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgjj\" (UniqueName: \"kubernetes.io/projected/58d74f63-ce55-4673-9752-584c1d65522f-kube-api-access-xjgjj\") pod \"ingress-canary-tvb95\" (UID: \"58d74f63-ce55-4673-9752-584c1d65522f\") " pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.643488 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.643460    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tvb95"
Jul 31 14:48:33.705414 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705377    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-config-volume\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.705577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705423    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.705577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705463    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-tmp-dir\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.705577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9kw\" (UniqueName: \"kubernetes.io/projected/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-kube-api-access-2p9kw\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.705726 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:33.705637    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:33.705726 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:33.705722    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:34.20569849 +0000 UTC m=+36.363949965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:33.705900 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705874    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-tmp-dir\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.706008 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.705986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-config-volume\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.720456 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.720398    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9kw\" (UniqueName: \"kubernetes.io/projected/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-kube-api-access-2p9kw\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:33.797080 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:33.796979    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tvb95"]
Jul 31 14:48:33.802058 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:33.802027    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d74f63_ce55_4673_9752_584c1d65522f.slice/crio-30f9777ff45a125c2d0f82cafea306a5cae439bf7c708e02ba09d82a5d526853 WatchSource:0}: Error finding container 30f9777ff45a125c2d0f82cafea306a5cae439bf7c708e02ba09d82a5d526853: Status 404 returned error can't find the container with id 30f9777ff45a125c2d0f82cafea306a5cae439bf7c708e02ba09d82a5d526853
Jul 31 14:48:34.008538 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.008497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:34.008740 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:34.008653    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:34.008740 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:34.008678    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:34.008740 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:34.008737    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:35.008721626 +0000 UTC m=+37.166973093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:34.209887 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.209851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:34.210065 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:34.210014    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:34.210122 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:34.210081    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:35.210066425 +0000 UTC m=+37.368317897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:34.242007 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.241976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:48:34.242178 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.241976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:48:34.242178 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.241976    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:48:34.247472 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247443    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\""
Jul 31 14:48:34.247472 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247468    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\""
Jul 31 14:48:34.247669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247489    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:48:34.247669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:48:34.247669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247447    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:48:34.247825 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.247789    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:48:34.467417 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:34.467307    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tvb95" event={"ID":"58d74f63-ce55-4673-9752-584c1d65522f","Type":"ContainerStarted","Data":"30f9777ff45a125c2d0f82cafea306a5cae439bf7c708e02ba09d82a5d526853"}
Jul 31 14:48:35.017269 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:35.017219    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:35.017459 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:35.017391    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:35.017459 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:35.017411    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:35.017580 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:35.017479    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:37.017457958 +0000 UTC m=+39.175709425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:35.218885 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:35.218846    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:35.219066 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:35.218990    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:35.219066 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:35.219052    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:37.21903575 +0000 UTC m=+39.377287223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:36.475622 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:36.475587    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="b27a45ada9886531d66a23f8b45611068eaedf3355cbc054871d5d359be17ffa" exitCode=0
Jul 31 14:48:36.476123 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:36.475672    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"b27a45ada9886531d66a23f8b45611068eaedf3355cbc054871d5d359be17ffa"}
Jul 31 14:48:37.033791 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.033753    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:37.033979 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:37.033959    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:37.034047 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:37.033984    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:37.034084 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:37.034056    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.034033986 +0000 UTC m=+43.192285458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:37.235002 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.234932    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:37.235114 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:37.235079    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:37.235155 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:37.235137    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.235122025 +0000 UTC m=+43.393373497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:37.479509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.479277    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tvb95" event={"ID":"58d74f63-ce55-4673-9752-584c1d65522f","Type":"ContainerStarted","Data":"5cee105ccce5d998f5e75f010edac213a4f9041ba50d77da1fa9e0f475ccd6fd"}
Jul 31 14:48:37.483758 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.482410    2557 generic.go:356] "Generic (PLEG): container finished" podID="b1b80295-cf3a-4204-9e2f-2703eef3aa50" containerID="6f83a68ab266bcac10c9c0f667830e892bd7de6acd2448c620dcf602cba070a9" exitCode=0
Jul 31 14:48:37.483758 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.482478    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerDied","Data":"6f83a68ab266bcac10c9c0f667830e892bd7de6acd2448c620dcf602cba070a9"}
Jul 31 14:48:37.564093 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:37.564029    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tvb95" podStartSLOduration=1.089082614 podStartE2EDuration="4.564006458s" totalImagesPullingTime="3.474923844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:37.507003909 +0000 UTC m=+39.665255401" watchObservedRunningTime="2026-07-31 14:48:37.564006458 +0000 UTC m=+39.722257948"
Jul 31 14:48:38.488588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:38.488551    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-znz82" event={"ID":"b1b80295-cf3a-4204-9e2f-2703eef3aa50","Type":"ContainerStarted","Data":"b79d69ba6e69ddd07d0e83a6583d7e40ec2a6e301c27cdca35e2389d2b7d14c9"}
Jul 31 14:48:38.532217 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:38.532166    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-znz82" podStartSLOduration=9.98289581 podStartE2EDuration="39.532148284s" totalImagesPullingTime="29.549252474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:38.529335996 +0000 UTC m=+40.687587487" watchObservedRunningTime="2026-07-31 14:48:38.532148284 +0000 UTC m=+40.690399775"
Jul 31 14:48:41.064674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:41.064631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:41.065053 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:41.064780    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:41.065053 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:41.064800    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:41.065053 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:41.064855    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:49.064840246 +0000 UTC m=+51.223091715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:41.266335 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:41.266293    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:41.266506 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:41.266415    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:41.266506 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:41.266465    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:48:49.266452614 +0000 UTC m=+51.424704082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:49.127597 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:49.127558    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:48:49.128071 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:49.127687    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:48:49.128071 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:49.127699    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:48:49.128071 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:49.127757    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.127742661 +0000 UTC m=+67.285994129 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:48:49.329574 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:49.329537    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:48:49.329743 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:49.329666    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:49.329743 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:49.329718    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.329705094 +0000 UTC m=+67.487956562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:53.419058 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.419019    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"]
Jul 31 14:48:53.421004 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.420983    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-wdlx9"]
Jul 31 14:48:53.421137 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.421115    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.422649 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.422632    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-7ltxn"]
Jul 31 14:48:53.422787 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.422768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.423717 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.423696    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-88vrx\""
Jul 31 14:48:53.423828 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.423737    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 14:48:53.424209 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.424190    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:48:53.424391 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.424376    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.425363 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425339    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:48:53.425478 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425352    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:48:53.425478 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425405    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 14:48:53.425680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425658    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:48:53.425680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425663    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 14:48:53.425775 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.425695    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8w7bj\""
Jul 31 14:48:53.426492 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.426475    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 14:48:53.428026 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.428010    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 14:48:53.428152 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.428134    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:53.428299 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.428268    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 14:48:53.428402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.428343    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-j8stk\""
Jul 31 14:48:53.428402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.428390    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:53.436493 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.436465    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 14:48:53.437417 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.437392    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 14:48:53.437657 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.437590    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"]
Jul 31 14:48:53.439023 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.439001    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wdlx9"]
Jul 31 14:48:53.440060 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.440042    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-7ltxn"]
Jul 31 14:48:53.457884 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.457859    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rbm\" (UniqueName: \"kubernetes.io/projected/f0a42421-6f22-42af-99a2-15f971f5c897-kube-api-access-56rbm\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.457969 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.457889    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-snapshots\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.457969 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.457906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-config\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.457969 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.457922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-trusted-ca\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.457975    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k942p\" (UniqueName: \"kubernetes.io/projected/b6504bf4-ba53-48c5-aa00-91d40893f55b-kube-api-access-k942p\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458034    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfc8t\" (UniqueName: \"kubernetes.io/projected/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-kube-api-access-wfc8t\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-trusted-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458064    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-service-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.458082 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458077    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.458313 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458151    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6504bf4-ba53-48c5-aa00-91d40893f55b-serving-cert\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.458313 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458176    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a42421-6f22-42af-99a2-15f971f5c897-serving-cert\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.458313 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.458192    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-tmp\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.559254 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559203    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rbm\" (UniqueName: \"kubernetes.io/projected/f0a42421-6f22-42af-99a2-15f971f5c897-kube-api-access-56rbm\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.559416 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-snapshots\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.559416 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559288    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-config\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.559416 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559311    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-trusted-ca\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.559535 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559468    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.559535 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559513    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k942p\" (UniqueName: \"kubernetes.io/projected/b6504bf4-ba53-48c5-aa00-91d40893f55b-kube-api-access-k942p\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.559614 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfc8t\" (UniqueName: \"kubernetes.io/projected/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-kube-api-access-wfc8t\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.559614 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559607    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-trusted-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.559716 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:53.559629    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:53.559716 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-service-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.559805 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:53.559741    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:54.059715387 +0000 UTC m=+56.217966861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:53.559805 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559777    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.559907 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559849    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6504bf4-ba53-48c5-aa00-91d40893f55b-serving-cert\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.559907 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559885    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a42421-6f22-42af-99a2-15f971f5c897-serving-cert\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560014 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.559915    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-tmp\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-snapshots\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560061    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-config\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.560254 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560214    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0a42421-6f22-42af-99a2-15f971f5c897-tmp\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560330 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560270    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-service-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560369 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560351    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6504bf4-ba53-48c5-aa00-91d40893f55b-trusted-ca\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.560475 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560456    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0a42421-6f22-42af-99a2-15f971f5c897-trusted-ca-bundle\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.560724 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.560701    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.563869 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.563837    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6504bf4-ba53-48c5-aa00-91d40893f55b-serving-cert\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.564004 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.563918    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f0a42421-6f22-42af-99a2-15f971f5c897-serving-cert\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.576166 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.576137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rbm\" (UniqueName: \"kubernetes.io/projected/f0a42421-6f22-42af-99a2-15f971f5c897-kube-api-access-56rbm\") pod \"insights-operator-77696685b-wdlx9\" (UID: \"f0a42421-6f22-42af-99a2-15f971f5c897\") " pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.576427 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.576410    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k942p\" (UniqueName: \"kubernetes.io/projected/b6504bf4-ba53-48c5-aa00-91d40893f55b-kube-api-access-k942p\") pod \"console-operator-5ccd6b445f-7ltxn\" (UID: \"b6504bf4-ba53-48c5-aa00-91d40893f55b\") " pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.576601 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.576581    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfc8t\" (UniqueName: \"kubernetes.io/projected/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-kube-api-access-wfc8t\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:53.743087 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.742998    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-wdlx9"
Jul 31 14:48:53.749287 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.749263    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:48:53.874903 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.874873    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-wdlx9"]
Jul 31 14:48:53.878646 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:53.878619    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a42421_6f22_42af_99a2_15f971f5c897.slice/crio-7ca4ab3b1e65142c8966229a8d219f9a866bcba1c17e3d6b4d075de32a35ffd6 WatchSource:0}: Error finding container 7ca4ab3b1e65142c8966229a8d219f9a866bcba1c17e3d6b4d075de32a35ffd6: Status 404 returned error can't find the container with id 7ca4ab3b1e65142c8966229a8d219f9a866bcba1c17e3d6b4d075de32a35ffd6
Jul 31 14:48:53.894577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:53.894549    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-7ltxn"]
Jul 31 14:48:53.897910 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:48:53.897883    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6504bf4_ba53_48c5_aa00_91d40893f55b.slice/crio-d1c444e280f9ab30e3615dd11ce083e3995e4872c80c18cf537b621d95ab7f68 WatchSource:0}: Error finding container d1c444e280f9ab30e3615dd11ce083e3995e4872c80c18cf537b621d95ab7f68: Status 404 returned error can't find the container with id d1c444e280f9ab30e3615dd11ce083e3995e4872c80c18cf537b621d95ab7f68
Jul 31 14:48:54.064429 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:54.064331    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:54.064578 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:54.064492    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:54.064578 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:54.064560    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:55.06454414 +0000 UTC m=+57.222795608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:54.530440 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:54.530399    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" event={"ID":"b6504bf4-ba53-48c5-aa00-91d40893f55b","Type":"ContainerStarted","Data":"d1c444e280f9ab30e3615dd11ce083e3995e4872c80c18cf537b621d95ab7f68"}
Jul 31 14:48:54.532514 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:54.532466    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wdlx9" event={"ID":"f0a42421-6f22-42af-99a2-15f971f5c897","Type":"ContainerStarted","Data":"7ca4ab3b1e65142c8966229a8d219f9a866bcba1c17e3d6b4d075de32a35ffd6"}
Jul 31 14:48:55.072806 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:55.072747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:55.073001 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:55.072919    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:55.073001 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:55.072985    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.072966513 +0000 UTC m=+59.231217983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:56.539139 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.539109    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/0.log"
Jul 31 14:48:56.539610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.539149    2557 generic.go:356] "Generic (PLEG): container finished" podID="b6504bf4-ba53-48c5-aa00-91d40893f55b" containerID="a0e085acc9276784a51aa72d3a481a8fac6777ced796660cd05a42a0e423ae98" exitCode=255
Jul 31 14:48:56.539610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.539184    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" event={"ID":"b6504bf4-ba53-48c5-aa00-91d40893f55b","Type":"ContainerDied","Data":"a0e085acc9276784a51aa72d3a481a8fac6777ced796660cd05a42a0e423ae98"}
Jul 31 14:48:56.539610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.539540    2557 scope.go:117] "RemoveContainer" containerID="a0e085acc9276784a51aa72d3a481a8fac6777ced796660cd05a42a0e423ae98"
Jul 31 14:48:56.540830 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.540804    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wdlx9" event={"ID":"f0a42421-6f22-42af-99a2-15f971f5c897","Type":"ContainerStarted","Data":"417ca96e2dd831ce779334cdf740e3e905bec4e5a0470b8fbaa738e3dbc7482f"}
Jul 31 14:48:56.574585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:56.574540    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-wdlx9" podStartSLOduration=1.336538415 podStartE2EDuration="3.574521749s" totalImagesPullingTime="2.237983334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:56.57360419 +0000 UTC m=+58.731855681" watchObservedRunningTime="2026-07-31 14:48:56.574521749 +0000 UTC m=+58.732773264"
Jul 31 14:48:57.091476 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.091419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:48:57.091687 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.091578    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:57.091687 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.091650    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.091634355 +0000 UTC m=+63.249885823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:48:57.375765 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.375694    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n2756"]
Jul 31 14:48:57.378630 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.378615    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.384186 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.384162    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\""
Jul 31 14:48:57.386173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.386149    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:48:57.386173 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.386164    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:48:57.393780 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.393755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ce77561-80c9-4f58-aba1-ebbe60e62527-crio-socket\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.393893 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.393794    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lm4\" (UniqueName: \"kubernetes.io/projected/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-api-access-m7lm4\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.393893 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.393869    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.393966 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.393894    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.393966 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.393941    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ce77561-80c9-4f58-aba1-ebbe60e62527-data-volume\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.402916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.402891    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2756"]
Jul 31 14:48:57.494409 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ce77561-80c9-4f58-aba1-ebbe60e62527-data-volume\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494431    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ce77561-80c9-4f58-aba1-ebbe60e62527-crio-socket\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lm4\" (UniqueName: \"kubernetes.io/projected/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-api-access-m7lm4\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494505    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494510    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ce77561-80c9-4f58-aba1-ebbe60e62527-crio-socket\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494610 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494827 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.494612    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:57.494827 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.494621    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:57.494827 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.494676    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.994655499 +0000 UTC m=+60.152906972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:57.494827 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.494687    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ce77561-80c9-4f58-aba1-ebbe60e62527-data-volume\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.494827 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.494726    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.994685373 +0000 UTC m=+60.152936847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:57.508002 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.507974    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lm4\" (UniqueName: \"kubernetes.io/projected/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-api-access-m7lm4\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:57.544423 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.544397    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/1.log"
Jul 31 14:48:57.544814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.544780    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/0.log"
Jul 31 14:48:57.544874 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.544811    2557 generic.go:356] "Generic (PLEG): container finished" podID="b6504bf4-ba53-48c5-aa00-91d40893f55b" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e" exitCode=255
Jul 31 14:48:57.544935 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.544908    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" event={"ID":"b6504bf4-ba53-48c5-aa00-91d40893f55b","Type":"ContainerDied","Data":"d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"}
Jul 31 14:48:57.545051 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.544979    2557 scope.go:117] "RemoveContainer" containerID="a0e085acc9276784a51aa72d3a481a8fac6777ced796660cd05a42a0e423ae98"
Jul 31 14:48:57.545209 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:57.545190    2557 scope.go:117] "RemoveContainer" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"
Jul 31 14:48:57.545454 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:57.545423    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-7ltxn_openshift-console-operator(b6504bf4-ba53-48c5-aa00-91d40893f55b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podUID="b6504bf4-ba53-48c5-aa00-91d40893f55b"
Jul 31 14:48:58.000233 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:58.000194    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:58.000233 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:58.000258    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:58.000517 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:58.000329    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:58.000517 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:58.000392    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:58.000517 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:58.000408    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.000389122 +0000 UTC m=+61.158640601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:58.000517 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:58.000433    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.000420999 +0000 UTC m=+61.158672487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:58.549339 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:58.549310    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/1.log"
Jul 31 14:48:58.549729 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:58.549624    2557 scope.go:117] "RemoveContainer" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"
Jul 31 14:48:58.549784 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:58.549766    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-7ltxn_openshift-console-operator(b6504bf4-ba53-48c5-aa00-91d40893f55b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podUID="b6504bf4-ba53-48c5-aa00-91d40893f55b"
Jul 31 14:48:58.739888 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:58.739860    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8ptv_88b37453-42ef-47fc-97fe-ad3c86bee43f/dns-node-resolver/0.log"
Jul 31 14:48:59.011007 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:59.010969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:59.011181 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:59.011017    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:48:59.011181 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:59.011094    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:48:59.011181 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:59.011161    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.011146309 +0000 UTC m=+63.169397776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : configmap "kube-rbac-proxy" not found
Jul 31 14:48:59.011340 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:59.011209    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:59.011340 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:48:59.011300    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.01128409 +0000 UTC m=+63.169535559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:48:59.463078 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:59.463049    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bgwc7"
Jul 31 14:48:59.749341 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:48:59.749268    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2k6ch_3b43d8aa-a300-479e-abab-9ef368b0df29/node-ca/0.log"
Jul 31 14:49:00.739400 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:00.739369    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tvb95_58d74f63-ce55-4673-9752-584c1d65522f/serve-healthcheck-canary/0.log"
Jul 31 14:49:01.030665 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:01.030557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:01.030665 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:01.030616    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:01.031079 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.030701    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:01.031079 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.030754    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:01.031079 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.030787    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.03076536 +0000 UTC m=+67.189016851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:01.031079 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.030856    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.030849694 +0000 UTC m=+67.189101161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:01.131532 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:01.131491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:49:01.131692 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.131633    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:01.131731 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:01.131698    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:09.131682563 +0000 UTC m=+71.289934032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:03.749455 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.749404    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:03.749455 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.749473    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:03.749887 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.749792    2557 scope.go:117] "RemoveContainer" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"
Jul 31 14:49:03.749952 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:03.749930    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-7ltxn_openshift-console-operator(b6504bf4-ba53-48c5-aa00-91d40893f55b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podUID="b6504bf4-ba53-48c5-aa00-91d40893f55b"
Jul 31 14:49:03.951882 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.951835    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:49:03.952091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.951986    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:49:03.955350 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.955332    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:49:03.955804 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.955790    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:03.964294 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.964268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/368ae851-9ae7-4d58-8715-aa95f856f689-original-pull-secret\") pod \"global-pull-secret-syncer-jhwrc\" (UID: \"368ae851-9ae7-4d58-8715-aa95f856f689\") " pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:49:03.964613 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.964598    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:49:03.975186 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:03.975153    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghjf\" (UniqueName: \"kubernetes.io/projected/00464f4b-81cb-4133-8f58-e017c9625482-kube-api-access-wghjf\") pod \"network-check-target-zbjsf\" (UID: \"00464f4b-81cb-4133-8f58-e017c9625482\") " pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:49:04.052782 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.052675    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:49:04.056500 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.056478    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:04.063695 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:04.063678    2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 14:49:04.063766 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:04.063742    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs podName:f1ad65b5-6b4d-4d71-8ca8-b9326a58b567 nodeName:}" failed. No retries permitted until 2026-07-31 14:50:08.063726584 +0000 UTC m=+130.221978053 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs") pod "network-metrics-daemon-6jf5r" (UID: "f1ad65b5-6b4d-4d71-8ca8-b9326a58b567") : secret "metrics-daemon-secret" not found
Jul 31 14:49:04.260479 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.260441    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jhwrc"
Jul 31 14:49:04.271306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.271285    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2kwb2\""
Jul 31 14:49:04.278213 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.278194    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:49:04.396923 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.396890    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jhwrc"]
Jul 31 14:49:04.400427 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:04.400400    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368ae851_9ae7_4d58_8715_aa95f856f689.slice/crio-c2703c34f3a96718cb361cbdd64e07dcdde2df6af3176c05c1877f879052ed63 WatchSource:0}: Error finding container c2703c34f3a96718cb361cbdd64e07dcdde2df6af3176c05c1877f879052ed63: Status 404 returned error can't find the container with id c2703c34f3a96718cb361cbdd64e07dcdde2df6af3176c05c1877f879052ed63
Jul 31 14:49:04.418919 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.418896    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zbjsf"]
Jul 31 14:49:04.421528 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:04.421504    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00464f4b_81cb_4133_8f58_e017c9625482.slice/crio-7c09c26f9d3c0327e241f3f6653b2a9d9e9cdf710d4f8c9e05b76b85dfc508e5 WatchSource:0}: Error finding container 7c09c26f9d3c0327e241f3f6653b2a9d9e9cdf710d4f8c9e05b76b85dfc508e5: Status 404 returned error can't find the container with id 7c09c26f9d3c0327e241f3f6653b2a9d9e9cdf710d4f8c9e05b76b85dfc508e5
Jul 31 14:49:04.563677 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.563593    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbjsf" event={"ID":"00464f4b-81cb-4133-8f58-e017c9625482","Type":"ContainerStarted","Data":"7c09c26f9d3c0327e241f3f6653b2a9d9e9cdf710d4f8c9e05b76b85dfc508e5"}
Jul 31 14:49:04.564756 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:04.564733    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jhwrc" event={"ID":"368ae851-9ae7-4d58-8715-aa95f856f689","Type":"ContainerStarted","Data":"c2703c34f3a96718cb361cbdd64e07dcdde2df6af3176c05c1877f879052ed63"}
Jul 31 14:49:05.061123 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:05.061085    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:05.061618 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:05.061149    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:05.061618 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.061353    2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:05.061618 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.061374    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:05.061618 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.061439    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:13.061415668 +0000 UTC m=+75.219667155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:05.061618 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.061463    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:13.061453073 +0000 UTC m=+75.219704556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:05.162111 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:05.162057    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:49:05.162311 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.162266    2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:05.162311 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.162287    2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-798f5c847c-rbbwd: secret "image-registry-tls" not found
Jul 31 14:49:05.162437 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.162361    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls podName:4cbd8171-cc73-4d74-84f6-967b722e3f75 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:37.162344358 +0000 UTC m=+99.320595825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls") pod "image-registry-798f5c847c-rbbwd" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75") : secret "image-registry-tls" not found
Jul 31 14:49:05.364043 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:05.363956    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:05.364221 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.364131    2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:05.364221 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:05.364209    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls podName:f90ef8e2-afdd-4440-8c3f-5d268609ba6e nodeName:}" failed. No retries permitted until 2026-07-31 14:49:37.364188995 +0000 UTC m=+99.522440468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls") pod "dns-default-lkpw6" (UID: "f90ef8e2-afdd-4440-8c3f-5d268609ba6e") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:09.197362 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.197318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:49:09.197763 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:09.197475    2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:09.197763 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:09.197547    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls podName:5a8fb483-e8a9-45e6-aabb-7ec50eb47049 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:25.19753231 +0000 UTC m=+87.355783778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-vbzzx" (UID: "5a8fb483-e8a9-45e6-aabb-7ec50eb47049") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:49:09.580349 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.580253    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jhwrc" event={"ID":"368ae851-9ae7-4d58-8715-aa95f856f689","Type":"ContainerStarted","Data":"7ec912c79611826dc9e5cc925a86d21fbdb2b79ef27aae058e90b8609fcdd96a"}
Jul 31 14:49:09.581690 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.581665    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zbjsf" event={"ID":"00464f4b-81cb-4133-8f58-e017c9625482","Type":"ContainerStarted","Data":"41ce1238147d028e599bc448b01eaaf7e8be3092743bd879cad3d1ed4b9a028d"}
Jul 31 14:49:09.581836 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.581792    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:49:09.597249 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.597179    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jhwrc" podStartSLOduration=66.189224824 podStartE2EDuration="1m10.597159679s" totalImagesPullingTime="4.407934855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:09.59701131 +0000 UTC m=+71.755262797" watchObservedRunningTime="2026-07-31 14:49:09.597159679 +0000 UTC m=+71.755411170"
Jul 31 14:49:09.613457 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:09.613403    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zbjsf" podStartSLOduration=66.22987371400001 podStartE2EDuration="1m10.613386172s" totalImagesPullingTime="4.383512458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:09.61316499 +0000 UTC m=+71.771416490" watchObservedRunningTime="2026-07-31 14:49:09.613386172 +0000 UTC m=+71.771637640"
Jul 31 14:49:13.129934 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:13.129887    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:13.130379 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:13.129947    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:13.130379 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:13.130130    2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:13.130379 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:13.130198    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls podName:4ce77561-80c9-4f58-aba1-ebbe60e62527 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.130182229 +0000 UTC m=+91.288433697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n2756" (UID: "4ce77561-80c9-4f58-aba1-ebbe60e62527") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:13.130490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:13.130420    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ce77561-80c9-4f58-aba1-ebbe60e62527-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:15.242311 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.242270    2557 scope.go:117] "RemoveContainer" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"
Jul 31 14:49:15.600216 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.600143    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:49:15.600572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.600558    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/1.log"
Jul 31 14:49:15.600631 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.600591    2557 generic.go:356] "Generic (PLEG): container finished" podID="b6504bf4-ba53-48c5-aa00-91d40893f55b" containerID="057e25d44355bf8872358ed08a6fc387a46918679473046e8d6324416a17ca24" exitCode=255
Jul 31 14:49:15.600680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.600661    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" event={"ID":"b6504bf4-ba53-48c5-aa00-91d40893f55b","Type":"ContainerDied","Data":"057e25d44355bf8872358ed08a6fc387a46918679473046e8d6324416a17ca24"}
Jul 31 14:49:15.600730 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.600720    2557 scope.go:117] "RemoveContainer" containerID="d1d0ece93370a2a897bb037931443f80d46601ce609cf770b521e905778a545e"
Jul 31 14:49:15.601077 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:15.601057    2557 scope.go:117] "RemoveContainer" containerID="057e25d44355bf8872358ed08a6fc387a46918679473046e8d6324416a17ca24"
Jul 31 14:49:15.601269 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:15.601229    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-7ltxn_openshift-console-operator(b6504bf4-ba53-48c5-aa00-91d40893f55b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podUID="b6504bf4-ba53-48c5-aa00-91d40893f55b"
Jul 31 14:49:16.604519 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:16.604494    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:49:23.750206 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:23.750054    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:23.750206 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:23.750132    2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:23.750790 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:23.750534    2557 scope.go:117] "RemoveContainer" containerID="057e25d44355bf8872358ed08a6fc387a46918679473046e8d6324416a17ca24"
Jul 31 14:49:23.750790 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:23.750731    2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-7ltxn_openshift-console-operator(b6504bf4-ba53-48c5-aa00-91d40893f55b)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podUID="b6504bf4-ba53-48c5-aa00-91d40893f55b"
Jul 31 14:49:25.236728 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.236692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:49:25.239092 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.239067    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a8fb483-e8a9-45e6-aabb-7ec50eb47049-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-vbzzx\" (UID: \"5a8fb483-e8a9-45e6-aabb-7ec50eb47049\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:49:25.535364 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.535285    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-88vrx\""
Jul 31 14:49:25.543777 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.543755    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"
Jul 31 14:49:25.626817 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.626784    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"]
Jul 31 14:49:25.635579 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.635506    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"]
Jul 31 14:49:25.636702 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.635788    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.642925 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.639820    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 31 14:49:25.642925 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.640911    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 31 14:49:25.642925 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.641394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 31 14:49:25.642925 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.641765    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 31 14:49:25.644710 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.644377    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.646941 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.646922    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-4rjhl\""
Jul 31 14:49:25.647052 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.647034    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 31 14:49:25.647130 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.647081    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 31 14:49:25.650777 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.650751    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"]
Jul 31 14:49:25.651888 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.651866    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"]
Jul 31 14:49:25.684541 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.684449    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx"]
Jul 31 14:49:25.687175 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:25.687148    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8fb483_e8a9_45e6_aabb_7ec50eb47049.slice/crio-1825976913ca846c62ddf290561ed034922d4387dc9dd9371f940b236fdcac18 WatchSource:0}: Error finding container 1825976913ca846c62ddf290561ed034922d4387dc9dd9371f940b236fdcac18: Status 404 returned error can't find the container with id 1825976913ca846c62ddf290561ed034922d4387dc9dd9371f940b236fdcac18
Jul 31 14:49:25.744374 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.744344    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f889f0eb-7573-4921-aa10-29bfccee7f37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.744374 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.744375    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9gv\" (UniqueName: \"kubernetes.io/projected/f889f0eb-7573-4921-aa10-29bfccee7f37-kube-api-access-rj9gv\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.744557 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.744482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.744557 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.744528    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f889f0eb-7573-4921-aa10-29bfccee7f37-tmp\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.744630 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.744575    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.845757 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.845672    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.845757 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.845742    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f889f0eb-7573-4921-aa10-29bfccee7f37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.845982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.845769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9gv\" (UniqueName: \"kubernetes.io/projected/f889f0eb-7573-4921-aa10-29bfccee7f37-kube-api-access-rj9gv\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.845982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.845859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.846086 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.846053    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f889f0eb-7573-4921-aa10-29bfccee7f37-tmp\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.846501 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.846475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f889f0eb-7573-4921-aa10-29bfccee7f37-tmp\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.846677 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.846656    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.848278 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.848258    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b80a5438-cc6d-4b63-ae98-7f10c04cdf4c-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-h5rrh\" (UID: \"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:25.848371 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.848301    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f889f0eb-7573-4921-aa10-29bfccee7f37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.856544 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.856523    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9gv\" (UniqueName: \"kubernetes.io/projected/f889f0eb-7573-4921-aa10-29bfccee7f37-kube-api-access-rj9gv\") pod \"klusterlet-addon-workmgr-5b79747579-kql5r\" (UID: \"f889f0eb-7573-4921-aa10-29bfccee7f37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.947859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.947819    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:25.956030 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:25.956005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"
Jul 31 14:49:26.084078 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:26.084050    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"]
Jul 31 14:49:26.087069 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:26.087044    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf889f0eb_7573_4921_aa10_29bfccee7f37.slice/crio-72e8efb13ca0841c3ec351949ba1e5d717443dfb04974d22bf2a7c08793fddd9 WatchSource:0}: Error finding container 72e8efb13ca0841c3ec351949ba1e5d717443dfb04974d22bf2a7c08793fddd9: Status 404 returned error can't find the container with id 72e8efb13ca0841c3ec351949ba1e5d717443dfb04974d22bf2a7c08793fddd9
Jul 31 14:49:26.136575 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:26.136489    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh"]
Jul 31 14:49:26.142988 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:26.142954    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb80a5438_cc6d_4b63_ae98_7f10c04cdf4c.slice/crio-48eb1b61b1dc735cb8db967971195c4c456e5f17a377efd2f068d21e3513ddfa WatchSource:0}: Error finding container 48eb1b61b1dc735cb8db967971195c4c456e5f17a377efd2f068d21e3513ddfa: Status 404 returned error can't find the container with id 48eb1b61b1dc735cb8db967971195c4c456e5f17a377efd2f068d21e3513ddfa
Jul 31 14:49:26.632045 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:26.631981    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r" event={"ID":"f889f0eb-7573-4921-aa10-29bfccee7f37","Type":"ContainerStarted","Data":"72e8efb13ca0841c3ec351949ba1e5d717443dfb04974d22bf2a7c08793fddd9"}
Jul 31 14:49:26.633463 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:26.633431    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx" event={"ID":"5a8fb483-e8a9-45e6-aabb-7ec50eb47049","Type":"ContainerStarted","Data":"1825976913ca846c62ddf290561ed034922d4387dc9dd9371f940b236fdcac18"}
Jul 31 14:49:26.634750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:26.634721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh" event={"ID":"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c","Type":"ContainerStarted","Data":"48eb1b61b1dc735cb8db967971195c4c456e5f17a377efd2f068d21e3513ddfa"}
Jul 31 14:49:29.176357 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:29.176314    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:29.178820 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:29.178791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ce77561-80c9-4f58-aba1-ebbe60e62527-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2756\" (UID: \"4ce77561-80c9-4f58-aba1-ebbe60e62527\") " pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:29.189705 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:29.189680    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v2m8k\""
Jul 31 14:49:29.197049 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:29.197029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2756"
Jul 31 14:49:30.100808 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.100768    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2756"]
Jul 31 14:49:30.104045 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:30.104001    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce77561_80c9_4f58_aba1_ebbe60e62527.slice/crio-5df315235f5e6357c28b160210df242259190cc3928c0f9b5e7414d0fc637959 WatchSource:0}: Error finding container 5df315235f5e6357c28b160210df242259190cc3928c0f9b5e7414d0fc637959: Status 404 returned error can't find the container with id 5df315235f5e6357c28b160210df242259190cc3928c0f9b5e7414d0fc637959
Jul 31 14:49:30.643481 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.643445    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"]
Jul 31 14:49:30.646529 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.646504    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:30.649635 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.649606    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 31 14:49:30.649798 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.649777    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5hjn8\""
Jul 31 14:49:30.652417 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.652376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2756" event={"ID":"4ce77561-80c9-4f58-aba1-ebbe60e62527","Type":"ContainerStarted","Data":"61b34d449a59d5abe2cc5ea2b489496cec19c430d8f81dbd74b20bc6e42fc94f"}
Jul 31 14:49:30.652527 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.652441    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2756" event={"ID":"4ce77561-80c9-4f58-aba1-ebbe60e62527","Type":"ContainerStarted","Data":"5df315235f5e6357c28b160210df242259190cc3928c0f9b5e7414d0fc637959"}
Jul 31 14:49:30.654387 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.654358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r" event={"ID":"f889f0eb-7573-4921-aa10-29bfccee7f37","Type":"ContainerStarted","Data":"78697a4b2e4150439e6acb95ae24a5b4eb5cc4332fbbff2acf53bf48b7409ac7"}
Jul 31 14:49:30.654562 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.654549    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:30.657811 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.657787    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"]
Jul 31 14:49:30.658198 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.658132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx" event={"ID":"5a8fb483-e8a9-45e6-aabb-7ec50eb47049","Type":"ContainerStarted","Data":"d3a84cf433d7c32f4114da5c5f154a26c22bec5b1b49d9c39d20dec59e9617fa"}
Jul 31 14:49:30.659050 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.659022    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r"
Jul 31 14:49:30.661260 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.661217    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh" event={"ID":"b80a5438-cc6d-4b63-ae98-7f10c04cdf4c","Type":"ContainerStarted","Data":"2dd3d30f9a373126e444f0fdddd49f41fc32dd0125fc26e476a1c89063cc5e65"}
Jul 31 14:49:30.690557 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.690478    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b79747579-kql5r" podStartSLOduration=1.7513870200000001 podStartE2EDuration="5.690441575s" totalImagesPullingTime="3.939054555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:30.689257061 +0000 UTC m=+92.847508550" watchObservedRunningTime="2026-07-31 14:49:30.690441575 +0000 UTC m=+92.848693058"
Jul 31 14:49:30.711408 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.711351    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-vbzzx" podStartSLOduration=33.453821583 podStartE2EDuration="37.711336476s" totalImagesPullingTime="4.257514893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:30.71054697 +0000 UTC m=+92.868798473" watchObservedRunningTime="2026-07-31 14:49:30.711336476 +0000 UTC m=+92.869587966"
Jul 31 14:49:30.734121 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.734073    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-h5rrh" podStartSLOduration=1.931727258 podStartE2EDuration="5.734057525s" totalImagesPullingTime="3.802330267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:30.732745115 +0000 UTC m=+92.890996606" watchObservedRunningTime="2026-07-31 14:49:30.734057525 +0000 UTC m=+92.892309038"
Jul 31 14:49:30.789058 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.789020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dq9h6\" (UID: \"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:30.890398 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:30.890372    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dq9h6\" (UID: \"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:30.890563 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:30.890548    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:30.890632 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:30.890609    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates podName:a2b394d2-edfb-4b26-8f18-e296e9f6dbbd nodeName:}" failed. No retries permitted until 2026-07-31 14:49:31.390592758 +0000 UTC m=+93.548844243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates") pod "prometheus-operator-admission-webhook-66685ff6d4-dq9h6" (UID: "a2b394d2-edfb-4b26-8f18-e296e9f6dbbd") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 31 14:49:31.393886 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:31.393839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dq9h6\" (UID: \"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:31.396508 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:31.396475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a2b394d2-edfb-4b26-8f18-e296e9f6dbbd-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-dq9h6\" (UID: \"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:31.560317 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:31.560269    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:31.667059 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:31.667007    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2756" event={"ID":"4ce77561-80c9-4f58-aba1-ebbe60e62527","Type":"ContainerStarted","Data":"5f108e4a254acbd3a7e4ce02e1c15dadaf34eaff589252ba60cabb27044f94da"}
Jul 31 14:49:31.708071 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:31.708048    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"]
Jul 31 14:49:32.028758 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:32.028725    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b394d2_edfb_4b26_8f18_e296e9f6dbbd.slice/crio-117f123c962ff55c85406fbf72d0218b3751adc9bbd408fbb9f0a8ff6f261363 WatchSource:0}: Error finding container 117f123c962ff55c85406fbf72d0218b3751adc9bbd408fbb9f0a8ff6f261363: Status 404 returned error can't find the container with id 117f123c962ff55c85406fbf72d0218b3751adc9bbd408fbb9f0a8ff6f261363
Jul 31 14:49:32.671767 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:32.671728    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6" event={"ID":"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd","Type":"ContainerStarted","Data":"117f123c962ff55c85406fbf72d0218b3751adc9bbd408fbb9f0a8ff6f261363"}
Jul 31 14:49:32.673675 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:32.673644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2756" event={"ID":"4ce77561-80c9-4f58-aba1-ebbe60e62527","Type":"ContainerStarted","Data":"4102b6c4410348231ff5e8a511e90d6af61def1f19edfad2e3e289826b1c2239"}
Jul 31 14:49:32.698777 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:32.698716    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n2756" podStartSLOduration=33.868014793 podStartE2EDuration="35.69869427s" totalImagesPullingTime="1.830679477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:32.698625281 +0000 UTC m=+94.856876771" watchObservedRunningTime="2026-07-31 14:49:32.69869427 +0000 UTC m=+94.856945761"
Jul 31 14:49:33.678232 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:33.678195    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6" event={"ID":"a2b394d2-edfb-4b26-8f18-e296e9f6dbbd","Type":"ContainerStarted","Data":"5d15743579003291311447a9518aa12da46e0e9db0f9b9e48d2fcb2f6f7f9e09"}
Jul 31 14:49:33.696008 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:33.695864    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6" podStartSLOduration=2.437720114 podStartE2EDuration="3.6958471s" totalImagesPullingTime="1.258126986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:33.695520655 +0000 UTC m=+95.853772146" watchObservedRunningTime="2026-07-31 14:49:33.6958471 +0000 UTC m=+95.854098593"
Jul 31 14:49:34.683787 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:34.683756    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:34.688484 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:34.688457    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-dq9h6"
Jul 31 14:49:35.716590 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.716560    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"]
Jul 31 14:49:35.719964 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.719947    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.722848 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.722828    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:35.722956 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.722826    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 31 14:49:35.723787 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.723769    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 31 14:49:35.724081 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.724065    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-srjxf\""
Jul 31 14:49:35.726981 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.726964    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.727062 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.726995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d598830-f534-4769-9200-7d04baee3a2d-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.727062 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.727056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.727169 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.727094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrcx\" (UniqueName: \"kubernetes.io/projected/8d598830-f534-4769-9200-7d04baee3a2d-kube-api-access-6qrcx\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.734330 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.734310    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"]
Jul 31 14:49:35.828215 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.828174    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrcx\" (UniqueName: \"kubernetes.io/projected/8d598830-f534-4769-9200-7d04baee3a2d-kube-api-access-6qrcx\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.828438 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.828275    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.828438 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.828309    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d598830-f534-4769-9200-7d04baee3a2d-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.828438 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.828371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.828605 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:35.828493    2557 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 31 14:49:35.828605 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:35.828564    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls podName:8d598830-f534-4769-9200-7d04baee3a2d nodeName:}" failed. No retries permitted until 2026-07-31 14:49:36.32854534 +0000 UTC m=+98.486796826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls") pod "prometheus-operator-758d7cb874-x9wc7" (UID: "8d598830-f534-4769-9200-7d04baee3a2d") : secret "prometheus-operator-tls" not found
Jul 31 14:49:35.829023 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.829003    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d598830-f534-4769-9200-7d04baee3a2d-metrics-client-ca\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.830590 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.830569    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:35.841180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:35.841157    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrcx\" (UniqueName: \"kubernetes.io/projected/8d598830-f534-4769-9200-7d04baee3a2d-kube-api-access-6qrcx\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:36.333634 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:36.333592    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:36.336014 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:36.335984    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d598830-f534-4769-9200-7d04baee3a2d-prometheus-operator-tls\") pod \"prometheus-operator-758d7cb874-x9wc7\" (UID: \"8d598830-f534-4769-9200-7d04baee3a2d\") " pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:36.628493 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:36.628402    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"
Jul 31 14:49:36.761445 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:36.761415    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-758d7cb874-x9wc7"]
Jul 31 14:49:36.764467 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:36.764435    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d598830_f534_4769_9200_7d04baee3a2d.slice/crio-082999719fccf20a9304f15a8ea054789f1dacca9c38eea0885b3bfe2d7cd53b WatchSource:0}: Error finding container 082999719fccf20a9304f15a8ea054789f1dacca9c38eea0885b3bfe2d7cd53b: Status 404 returned error can't find the container with id 082999719fccf20a9304f15a8ea054789f1dacca9c38eea0885b3bfe2d7cd53b
Jul 31 14:49:37.241560 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.241519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:49:37.241937 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.241920    2557 scope.go:117] "RemoveContainer" containerID="057e25d44355bf8872358ed08a6fc387a46918679473046e8d6324416a17ca24"
Jul 31 14:49:37.243961 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.243933    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"image-registry-798f5c847c-rbbwd\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") " pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:49:37.443756 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.443723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:37.445893 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.445868    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f90ef8e2-afdd-4440-8c3f-5d268609ba6e-metrics-tls\") pod \"dns-default-lkpw6\" (UID: \"f90ef8e2-afdd-4440-8c3f-5d268609ba6e\") " pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:37.518430 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.518339    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-plk7v\""
Jul 31 14:49:37.526150 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.526113    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:49:37.653615 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.653589    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p6mpf\""
Jul 31 14:49:37.660880 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.660852    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:37.697515 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.697483    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:49:37.698092 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.698074    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:49:37.698212 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.698190    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" event={"ID":"b6504bf4-ba53-48c5-aa00-91d40893f55b","Type":"ContainerStarted","Data":"35e99a5fda501c2a8acc42f528e3e8518f700fc4a734aeedc748d2c73118ccc5"}
Jul 31 14:49:37.699074 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.698887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:37.700609 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.700582    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7" event={"ID":"8d598830-f534-4769-9200-7d04baee3a2d","Type":"ContainerStarted","Data":"082999719fccf20a9304f15a8ea054789f1dacca9c38eea0885b3bfe2d7cd53b"}
Jul 31 14:49:37.702002 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:37.701802    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbd8171_cc73_4d74_84f6_967b722e3f75.slice/crio-b5aaac31ecbb6abdd6e6043e2c7b682e656cbedd6efbc19ae431d3ca2de15c5c WatchSource:0}: Error finding container b5aaac31ecbb6abdd6e6043e2c7b682e656cbedd6efbc19ae431d3ca2de15c5c: Status 404 returned error can't find the container with id b5aaac31ecbb6abdd6e6043e2c7b682e656cbedd6efbc19ae431d3ca2de15c5c
Jul 31 14:49:37.721421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.720443    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn" podStartSLOduration=42.499206889 podStartE2EDuration="44.720421554s" totalImagesPullingTime="2.221214665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:37.719536555 +0000 UTC m=+99.877788046" watchObservedRunningTime="2026-07-31 14:49:37.720421554 +0000 UTC m=+99.878673045"
Jul 31 14:49:37.824394 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:37.824367    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lkpw6"]
Jul 31 14:49:37.827963 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:37.827924    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90ef8e2_afdd_4440_8c3f_5d268609ba6e.slice/crio-7f0415b454c5839922fda9eb2afc54fda63abe7c157c0550a733fc8ef3fd8c57 WatchSource:0}: Error finding container 7f0415b454c5839922fda9eb2afc54fda63abe7c157c0550a733fc8ef3fd8c57: Status 404 returned error can't find the container with id 7f0415b454c5839922fda9eb2afc54fda63abe7c157c0550a733fc8ef3fd8c57
Jul 31 14:49:38.248335 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.248292    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-7ltxn"
Jul 31 14:49:38.452687 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.452655    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-wn6s5"]
Jul 31 14:49:38.457442 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.457379    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:49:38.461337 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.461115    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 14:49:38.461337 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.461332    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2tr6l\""
Jul 31 14:49:38.461889 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.461869    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 14:49:38.472090 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.472055    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wn6s5"]
Jul 31 14:49:38.556253 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.556148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkswt\" (UniqueName: \"kubernetes.io/projected/33af7aec-f40c-4baa-bed3-0b02839031e2-kube-api-access-nkswt\") pod \"downloads-77c974ffcf-wn6s5\" (UID: \"33af7aec-f40c-4baa-bed3-0b02839031e2\") " pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:49:38.656941 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.656906    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkswt\" (UniqueName: \"kubernetes.io/projected/33af7aec-f40c-4baa-bed3-0b02839031e2-kube-api-access-nkswt\") pod \"downloads-77c974ffcf-wn6s5\" (UID: \"33af7aec-f40c-4baa-bed3-0b02839031e2\") " pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:49:38.673197 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.673163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkswt\" (UniqueName: \"kubernetes.io/projected/33af7aec-f40c-4baa-bed3-0b02839031e2-kube-api-access-nkswt\") pod \"downloads-77c974ffcf-wn6s5\" (UID: \"33af7aec-f40c-4baa-bed3-0b02839031e2\") " pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:49:38.706917 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.706856    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7" event={"ID":"8d598830-f534-4769-9200-7d04baee3a2d","Type":"ContainerStarted","Data":"b6caeaea3f80d5ba961f627bef0d016aa57c7614b7d8d82b093320604fe4b810"}
Jul 31 14:49:38.707146 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.706939    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7" event={"ID":"8d598830-f534-4769-9200-7d04baee3a2d","Type":"ContainerStarted","Data":"00255f65f5a337272cfb06860d8d32b81b9a879aa14a5e2af2fd88cd1675853f"}
Jul 31 14:49:38.709269 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.708557    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkpw6" event={"ID":"f90ef8e2-afdd-4440-8c3f-5d268609ba6e","Type":"ContainerStarted","Data":"7f0415b454c5839922fda9eb2afc54fda63abe7c157c0550a733fc8ef3fd8c57"}
Jul 31 14:49:38.711641 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.711612    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" event={"ID":"4cbd8171-cc73-4d74-84f6-967b722e3f75","Type":"ContainerStarted","Data":"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"}
Jul 31 14:49:38.711767 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.711669    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" event={"ID":"4cbd8171-cc73-4d74-84f6-967b722e3f75","Type":"ContainerStarted","Data":"b5aaac31ecbb6abdd6e6043e2c7b682e656cbedd6efbc19ae431d3ca2de15c5c"}
Jul 31 14:49:38.734549 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.734501    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-758d7cb874-x9wc7" podStartSLOduration=2.156859955 podStartE2EDuration="3.734486139s" totalImagesPullingTime="1.577626184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:38.731492818 +0000 UTC m=+100.889744309" watchObservedRunningTime="2026-07-31 14:49:38.734486139 +0000 UTC m=+100.892737626"
Jul 31 14:49:38.772024 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.771973    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" podStartSLOduration=99.771956114 podStartE2EDuration="1m39.771956114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:38.771479608 +0000 UTC m=+100.929731098" watchObservedRunningTime="2026-07-31 14:49:38.771956114 +0000 UTC m=+100.930207618"
Jul 31 14:49:38.785593 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.785543    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:49:38.929597 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:38.929562    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-wn6s5"]
Jul 31 14:49:38.933039 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:38.932999    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33af7aec_f40c_4baa_bed3_0b02839031e2.slice/crio-2fb6eed498334b42c5e19d16d82b476c96e992c1a8265e2c9a69be27b394341d WatchSource:0}: Error finding container 2fb6eed498334b42c5e19d16d82b476c96e992c1a8265e2c9a69be27b394341d: Status 404 returned error can't find the container with id 2fb6eed498334b42c5e19d16d82b476c96e992c1a8265e2c9a69be27b394341d
Jul 31 14:49:39.717306 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:39.717269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkpw6" event={"ID":"f90ef8e2-afdd-4440-8c3f-5d268609ba6e","Type":"ContainerStarted","Data":"4598bb05f33d49c4d1fa0ddf3ce7f26734187b854762bb44cff904b2bd2aa59a"}
Jul 31 14:49:39.718552 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:39.718524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wn6s5" event={"ID":"33af7aec-f40c-4baa-bed3-0b02839031e2","Type":"ContainerStarted","Data":"2fb6eed498334b42c5e19d16d82b476c96e992c1a8265e2c9a69be27b394341d"}
Jul 31 14:49:39.718998 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:39.718956    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:49:40.586608 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:40.586576    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zbjsf"
Jul 31 14:49:40.726019 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:40.725923    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lkpw6" event={"ID":"f90ef8e2-afdd-4440-8c3f-5d268609ba6e","Type":"ContainerStarted","Data":"2009be9a0881553a0e00fc00b61b1c808f92d63bc9591e73e9aca1856b43e6c3"}
Jul 31 14:49:40.726282 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:40.726258    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:41.251921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.251860    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lkpw6" podStartSLOduration=66.467429617 podStartE2EDuration="1m8.251840914s" totalImagesPullingTime="1.784411297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:40.749029126 +0000 UTC m=+102.907280618" watchObservedRunningTime="2026-07-31 14:49:41.251840914 +0000 UTC m=+103.410092434"
Jul 31 14:49:41.252617 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.252593    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lt8hw"]
Jul 31 14:49:41.270563 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.270523    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.274169 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.274140    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:49:41.274781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.274524    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:49:41.274781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.274542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:49:41.274781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.274598    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzwk\""
Jul 31 14:49:41.380821 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380787    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.380983 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380843    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-textfile\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.380983 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380892    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-tls\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.380983 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380914    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-sys\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.380983 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.380983 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.380978    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-metrics-client-ca\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.381230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.381008    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-root\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.381230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.381030    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-wtmp\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.381230 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.381050    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjvt\" (UniqueName: \"kubernetes.io/projected/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-kube-api-access-sfjvt\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482472 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482420    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482605    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-textfile\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482670    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-tls\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482703    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-sys\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482808    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-metrics-client-ca\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.482859 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482842    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-root\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483183 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482866    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-wtmp\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483183 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.482892    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjvt\" (UniqueName: \"kubernetes.io/projected/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-kube-api-access-sfjvt\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483183 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483074    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-textfile\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483183 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483135    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-accelerators-collector-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483220    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-sys\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-root\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-wtmp\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.483598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.483577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-metrics-client-ca\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.485721 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.485693    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-tls\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.485819 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.485745    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.506049 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.505992    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjvt\" (UniqueName: \"kubernetes.io/projected/d6c3e29d-3c0b-47cc-802e-722d3697e9b8-kube-api-access-sfjvt\") pod \"node-exporter-lt8hw\" (UID: \"d6c3e29d-3c0b-47cc-802e-722d3697e9b8\") " pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.582421 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.582384    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lt8hw"
Jul 31 14:49:41.607981 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:41.607942    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c3e29d_3c0b_47cc_802e_722d3697e9b8.slice/crio-753c4134ab3698eef323b17fe37e12fced7f45905a951035f75e16f0a8cb5563 WatchSource:0}: Error finding container 753c4134ab3698eef323b17fe37e12fced7f45905a951035f75e16f0a8cb5563: Status 404 returned error can't find the container with id 753c4134ab3698eef323b17fe37e12fced7f45905a951035f75e16f0a8cb5563
Jul 31 14:49:41.731605 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:41.731517    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lt8hw" event={"ID":"d6c3e29d-3c0b-47cc-802e-722d3697e9b8","Type":"ContainerStarted","Data":"753c4134ab3698eef323b17fe37e12fced7f45905a951035f75e16f0a8cb5563"}
Jul 31 14:49:42.206986 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.206952    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:49:42.231572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.231531    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:49:42.231748 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.231634    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.236451 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.236423    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 14:49:42.236662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.236641    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 14:49:42.236726 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.236662    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tczp9\""
Jul 31 14:49:42.236840 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.236824    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 14:49:42.236903 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.236872    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 14:49:42.237096 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.237080    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 14:49:42.237231 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.237210    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 14:49:42.237708 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.237517    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 14:49:42.237819 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.237734    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 14:49:42.238337 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.238317    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291091    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t675w\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291150    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291187    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291221    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291284    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291430    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291468    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.291736 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.291576    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393476 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393438    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393527    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t675w\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393631    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393678 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393750    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393774    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393796    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393821    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.393990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.393836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.396690 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.395254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.396690 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:42.396146    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle podName:9aac8588-6d8b-4de5-b892-3c0a6525d5e2 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:42.896121324 +0000 UTC m=+105.054372794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2") : configmap references non-existent config key: ca-bundle.crt
Jul 31 14:49:42.397084 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:42.396979    2557 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 31 14:49:42.397084 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.397012    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.397084 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:49:42.397053    2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls podName:9aac8588-6d8b-4de5-b892-3c0a6525d5e2 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:42.897035842 +0000 UTC m=+105.055287312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2") : secret "alertmanager-main-tls" not found
Jul 31 14:49:42.397482 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.397457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.399253 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.399193    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.399807 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.399758    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.399973 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.399939    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.400387 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.400365    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.401510 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.401459    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.403592 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.403570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.404160 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.404128    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.404728 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.404702    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t675w\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.897500 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.897465    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.898009 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.897574    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.898382 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.898358    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:42.900596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:42.900569    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:43.144552 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:43.144510    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:49:43.464614 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:43.464580    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:49:43.468933 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:43.468903    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aac8588_6d8b_4de5_b892_3c0a6525d5e2.slice/crio-a7f403d4e841282cff18acbef402a736952e17d4ee0dd30f9056a0e514fbb9bd WatchSource:0}: Error finding container a7f403d4e841282cff18acbef402a736952e17d4ee0dd30f9056a0e514fbb9bd: Status 404 returned error can't find the container with id a7f403d4e841282cff18acbef402a736952e17d4ee0dd30f9056a0e514fbb9bd
Jul 31 14:49:43.743025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:43.742930    2557 generic.go:356] "Generic (PLEG): container finished" podID="d6c3e29d-3c0b-47cc-802e-722d3697e9b8" containerID="4654bf6ced20cbaba54280e2b994229b875ade643202ffc9e632bc2a661b0a8b" exitCode=0
Jul 31 14:49:43.743193 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:43.743030    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lt8hw" event={"ID":"d6c3e29d-3c0b-47cc-802e-722d3697e9b8","Type":"ContainerDied","Data":"4654bf6ced20cbaba54280e2b994229b875ade643202ffc9e632bc2a661b0a8b"}
Jul 31 14:49:43.744523 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:43.744489    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"a7f403d4e841282cff18acbef402a736952e17d4ee0dd30f9056a0e514fbb9bd"}
Jul 31 14:49:44.752522 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:44.752273    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lt8hw" event={"ID":"d6c3e29d-3c0b-47cc-802e-722d3697e9b8","Type":"ContainerStarted","Data":"7ad9afdd6594ff02bf18f5ff5ea962870799b7fb20ef2d3e9d82c63ef52a2079"}
Jul 31 14:49:44.752522 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:44.752345    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lt8hw" event={"ID":"d6c3e29d-3c0b-47cc-802e-722d3697e9b8","Type":"ContainerStarted","Data":"fb6cb8f9806e250a16eade42f1e917355bc4b13bf4c74e5025a6e23df1cb4a17"}
Jul 31 14:49:44.774109 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:44.774053    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lt8hw" podStartSLOduration=1.976453437 podStartE2EDuration="3.77403003s" totalImagesPullingTime="1.692111487s" totalInitContainerRuntime="105.465106ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:44.773327275 +0000 UTC m=+106.931578768" watchObservedRunningTime="2026-07-31 14:49:44.77403003 +0000 UTC m=+106.932281521"
Jul 31 14:49:45.759123 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.759085    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb" exitCode=0
Jul 31 14:49:45.759620 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.759177    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb"}
Jul 31 14:49:45.912748 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.911438    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"]
Jul 31 14:49:45.916177 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.916147    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:45.918778 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.918697    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 31 14:49:45.918982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.918819    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zc6kr\""
Jul 31 14:49:45.926666 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:45.926639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"]
Jul 31 14:49:46.028124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.028015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2379fd1-ed3c-4629-b028-ab394a084506-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.028124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.028074    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplqg\" (UniqueName: \"kubernetes.io/projected/d2379fd1-ed3c-4629-b028-ab394a084506-kube-api-access-jplqg\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.128849 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.128804    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2379fd1-ed3c-4629-b028-ab394a084506-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.129041 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.128865    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplqg\" (UniqueName: \"kubernetes.io/projected/d2379fd1-ed3c-4629-b028-ab394a084506-kube-api-access-jplqg\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.131860 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.131832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d2379fd1-ed3c-4629-b028-ab394a084506-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.141494 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.141460    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplqg\" (UniqueName: \"kubernetes.io/projected/d2379fd1-ed3c-4629-b028-ab394a084506-kube-api-access-jplqg\") pod \"monitoring-plugin-6dffbbd67-85m9f\" (UID: \"d2379fd1-ed3c-4629-b028-ab394a084506\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.227390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.227361    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:46.375285 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.375203    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"]
Jul 31 14:49:46.377843 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:46.377803    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2379fd1_ed3c_4629_b028_ab394a084506.slice/crio-45aff433b3b6a334f40d394b4660d4cff25936469c8c8941f75c262b04d4c143 WatchSource:0}: Error finding container 45aff433b3b6a334f40d394b4660d4cff25936469c8c8941f75c262b04d4c143: Status 404 returned error can't find the container with id 45aff433b3b6a334f40d394b4660d4cff25936469c8c8941f75c262b04d4c143
Jul 31 14:49:46.765501 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:46.765464    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f" event={"ID":"d2379fd1-ed3c-4629-b028-ab394a084506","Type":"ContainerStarted","Data":"45aff433b3b6a334f40d394b4660d4cff25936469c8c8941f75c262b04d4c143"}
Jul 31 14:49:47.539912 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.539882    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:49:47.544414 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.544388    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.545489 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.545456    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:49:47.549888 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.549865    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.550401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.550039    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 14:49:47.551674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.550894    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 14:49:47.551674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.550987    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 14:49:47.551674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.551093    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 14:49:47.551674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.551176    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-g2qkk\""
Jul 31 14:49:47.551674 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.551403    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 14:49:47.558278 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.558081    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 14:49:47.558963 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.558935    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 14:49:47.559109 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.558935    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 14:49:47.559365 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.559292    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6q4cb\""
Jul 31 14:49:47.559627 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.559608    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 14:49:47.559713 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.559639    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 14:49:47.559867 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.559851    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 14:49:47.559924 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.559873    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 14:49:47.560956 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.560014    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 14:49:47.560956 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.560153    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 14:49:47.560956 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.560752    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 14:49:47.561432 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.561256    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 14:49:47.561523 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.561433    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:49:47.569916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.566505    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4n5funqrsc9d3\""
Jul 31 14:49:47.569916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.569226    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 14:49:47.569916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.569486    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 14:49:47.569916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.569840    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 14:49:47.571463 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.571443    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 14:49:47.582542 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.582516    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:49:47.643757 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643712    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.643950 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643793    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.643950 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.643950 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2g5\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.643950 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643944    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.643984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644007    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644029    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644063    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644086    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644175 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644110    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644315    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644387    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644410    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644448    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644472    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644497    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644521    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8mj\" (UniqueName: \"kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644604    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644646    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644700    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644737    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644772    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.644814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.644800    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746394 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746411    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746450    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746492    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746527    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.746585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746555    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2g5\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746585    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746623    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746659    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746722    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746827    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746851    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.746892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746883    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746902    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746935    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746964    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.746982    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8mj\" (UniqueName: \"kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747027    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747062    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747119    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747155    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747163    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747333    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.747995 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.747856    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.748047 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.748007    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.748577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.748161    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.748577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.748299    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.751140 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.749892    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.751140 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.750441    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.751140 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.750519    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.751582 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.751402    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.751931 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.751907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.754069 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.754027    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.754810 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.754784    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.755193 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.755169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.755472 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.755442    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.756317 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.756203    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.758740 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.758702    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.759814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.759763    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.761565 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.761478    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.762366 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.761697    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.762366 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.761870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.762366 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.761989    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.763174 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.763131    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.763741 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.763718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2g5\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.764453 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.764343    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8mj\" (UniqueName: \"kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj\") pod \"console-564cb897f6-v8gwp\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") " pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.764540 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.764457    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:47.862706 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.862563    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:49:47.872804 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:47.872768    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:49:48.085616 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.085469    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:49:48.096452 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:48.096411    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5ad9e1_e995_4d0c_a141_9f468e2f6cb3.slice/crio-1ae52a69cab6491d8c419c288a65813311407680d4bc35ad47bd632706180e4a WatchSource:0}: Error finding container 1ae52a69cab6491d8c419c288a65813311407680d4bc35ad47bd632706180e4a: Status 404 returned error can't find the container with id 1ae52a69cab6491d8c419c288a65813311407680d4bc35ad47bd632706180e4a
Jul 31 14:49:48.119251 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.119113    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:49:48.129166 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:49:48.129121    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7cc88e_164a_4506_944d_22675354a3c9.slice/crio-188241d38dcf1f1a6229a099254771f3b1c9d6162b776b43a5283ba20b62ac9f WatchSource:0}: Error finding container 188241d38dcf1f1a6229a099254771f3b1c9d6162b776b43a5283ba20b62ac9f: Status 404 returned error can't find the container with id 188241d38dcf1f1a6229a099254771f3b1c9d6162b776b43a5283ba20b62ac9f
Jul 31 14:49:48.139219 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.139193    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:49:48.777014 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.776970    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cb897f6-v8gwp" event={"ID":"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3","Type":"ContainerStarted","Data":"1ae52a69cab6491d8c419c288a65813311407680d4bc35ad47bd632706180e4a"}
Jul 31 14:49:48.780877 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.780832    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"}
Jul 31 14:49:48.781010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.780932    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"}
Jul 31 14:49:48.781010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.780973    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"}
Jul 31 14:49:48.781127 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.781013    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"}
Jul 31 14:49:48.781127 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.781050    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"}
Jul 31 14:49:48.782621 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.782596    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f" event={"ID":"d2379fd1-ed3c-4629-b028-ab394a084506","Type":"ContainerStarted","Data":"53ecd116718b044286751d8cbea1d0768acb60188865a6483a0f14aab377768f"}
Jul 31 14:49:48.783661 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.783639    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:48.786380 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.785511    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351" exitCode=0
Jul 31 14:49:48.786380 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.785582    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351"}
Jul 31 14:49:48.786380 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.785641    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"188241d38dcf1f1a6229a099254771f3b1c9d6162b776b43a5283ba20b62ac9f"}
Jul 31 14:49:48.789823 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.789774    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f"
Jul 31 14:49:48.801731 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:48.801563    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-85m9f" podStartSLOduration=2.285893685 podStartE2EDuration="3.801543538s" totalImagesPullingTime="1.515649853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:48.79952303 +0000 UTC m=+110.957774521" watchObservedRunningTime="2026-07-31 14:49:48.801543538 +0000 UTC m=+110.959795029"
Jul 31 14:49:49.797562 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:49.797429    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerStarted","Data":"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"}
Jul 31 14:49:49.833545 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:49.832537    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.410850372 podStartE2EDuration="7.832515657s" totalImagesPullingTime="4.404395638s" totalInitContainerRuntime="17.269647ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:49.830425172 +0000 UTC m=+111.988676666" watchObservedRunningTime="2026-07-31 14:49:49.832515657 +0000 UTC m=+111.990767148"
Jul 31 14:49:50.734119 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:50.734063    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lkpw6"
Jul 31 14:49:58.147293 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:49:58.147261    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:50:00.849982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.849886    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"}
Jul 31 14:50:00.850507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.850017    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"}
Jul 31 14:50:00.851903 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.851871    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cb897f6-v8gwp" event={"ID":"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3","Type":"ContainerStarted","Data":"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"}
Jul 31 14:50:00.853619 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.853592    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-wn6s5" event={"ID":"33af7aec-f40c-4baa-bed3-0b02839031e2","Type":"ContainerStarted","Data":"d6c74f6b536bebcccaefb7945c025f6d02c96908944a1e304cb1b9ec1db44a18"}
Jul 31 14:50:00.853898 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.853874    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:50:00.868833 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.868799    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-wn6s5"
Jul 31 14:50:00.876979 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.876920    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-564cb897f6-v8gwp" podStartSLOduration=1.616507454 podStartE2EDuration="13.876900487s" totalImagesPullingTime="12.260393033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:00.874851859 +0000 UTC m=+123.033103364" watchObservedRunningTime="2026-07-31 14:50:00.876900487 +0000 UTC m=+123.035151978"
Jul 31 14:50:00.895918 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:00.895844    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-wn6s5" podStartSLOduration=1.481854145 podStartE2EDuration="22.895822146s" totalImagesPullingTime="21.413968001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:00.894133268 +0000 UTC m=+123.052384759" watchObservedRunningTime="2026-07-31 14:50:00.895822146 +0000 UTC m=+123.054073637"
Jul 31 14:50:03.873534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:03.873457    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"}
Jul 31 14:50:03.874016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:03.873561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"}
Jul 31 14:50:03.874016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:03.873608    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"}
Jul 31 14:50:03.874016 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:03.873636    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerStarted","Data":"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"}
Jul 31 14:50:03.911669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:03.911598    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.6244048490000003 podStartE2EDuration="16.91157618s" totalImagesPullingTime="14.26015728s" totalInitContainerRuntime="27.014051ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:03.909897165 +0000 UTC m=+126.068148696" watchObservedRunningTime="2026-07-31 14:50:03.91157618 +0000 UTC m=+126.069827673"
Jul 31 14:50:05.343412 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:05.343379    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:50:07.862856 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:07.862820    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:50:07.874013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:07.873975    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:08.066843 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.066793    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:50:08.069420 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.069392    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1ad65b5-6b4d-4d71-8ca8-b9326a58b567-metrics-certs\") pod \"network-metrics-daemon-6jf5r\" (UID: \"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567\") " pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:50:08.156218 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.156133    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qnc55\""
Jul 31 14:50:08.163508 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.163464    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jf5r"
Jul 31 14:50:08.355336 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.355301    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jf5r"]
Jul 31 14:50:08.358742 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:50:08.358697    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ad65b5_6b4d_4d71_8ca8_b9326a58b567.slice/crio-0214f64889ee685e50e0fd9739259e3d2025dcaa67b854b8ad354f41da3c4aae WatchSource:0}: Error finding container 0214f64889ee685e50e0fd9739259e3d2025dcaa67b854b8ad354f41da3c4aae: Status 404 returned error can't find the container with id 0214f64889ee685e50e0fd9739259e3d2025dcaa67b854b8ad354f41da3c4aae
Jul 31 14:50:08.899585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:08.899542    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jf5r" event={"ID":"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567","Type":"ContainerStarted","Data":"0214f64889ee685e50e0fd9739259e3d2025dcaa67b854b8ad354f41da3c4aae"}
Jul 31 14:50:10.911913 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:10.911877    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jf5r" event={"ID":"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567","Type":"ContainerStarted","Data":"9cabacd8ae574d36b66b10169592a530324ea3584eb2e1f3eb0a57cf1fbbf48c"}
Jul 31 14:50:10.912519 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:10.911940    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jf5r" event={"ID":"f1ad65b5-6b4d-4d71-8ca8-b9326a58b567","Type":"ContainerStarted","Data":"0c21113e8007883f90dd71c80649fe55f64bc3cc13e18b25df2b64936fe11833"}
Jul 31 14:50:10.928816 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:10.928767    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6jf5r" podStartSLOduration=130.518775269 podStartE2EDuration="2m11.928750638s" totalImagesPullingTime="1.409975369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:10.927555356 +0000 UTC m=+133.085806847" watchObservedRunningTime="2026-07-31 14:50:10.928750638 +0000 UTC m=+133.087002127"
Jul 31 14:50:13.170113 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.170057    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" podUID="4cbd8171-cc73-4d74-84f6-967b722e3f75" containerName="registry" containerID="cri-o://b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46" gracePeriod=30
Jul 31 14:50:13.471125 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.471100    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:50:13.514648 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514612    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514823 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514667    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514823 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514699    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514823 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514790    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj2ch\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514841    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514900    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.514989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.514928    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates\") pod \"4cbd8171-cc73-4d74-84f6-967b722e3f75\" (UID: \"4cbd8171-cc73-4d74-84f6-967b722e3f75\") "
Jul 31 14:50:13.515732 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.515673    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:13.516120 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.516095    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:13.518139 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.517960    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:13.518579 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.518206    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:13.518579 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.518420    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:13.518697 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.518599    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:13.518697 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.518602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch" (OuterVolumeSpecName: "kube-api-access-nj2ch") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "kube-api-access-nj2ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:13.524699 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.524555    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4cbd8171-cc73-4d74-84f6-967b722e3f75" (UID: "4cbd8171-cc73-4d74-84f6-967b722e3f75"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:13.616189 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616154    2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-image-registry-private-configuration\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616189 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616183    2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4cbd8171-cc73-4d74-84f6-967b722e3f75-ca-trust-extracted\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616189 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616194    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2ch\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-kube-api-access-nj2ch\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616202    2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-trusted-ca\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616211    2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4cbd8171-cc73-4d74-84f6-967b722e3f75-installation-pull-secrets\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616219    2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-bound-sa-token\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616228    2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-certificates\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.616450 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.616255    2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4cbd8171-cc73-4d74-84f6-967b722e3f75-registry-tls\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:13.925070 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.925035    2557 generic.go:356] "Generic (PLEG): container finished" podID="4cbd8171-cc73-4d74-84f6-967b722e3f75" containerID="b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46" exitCode=0
Jul 31 14:50:13.925271 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.925080    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd"
Jul 31 14:50:13.925271 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.925099    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" event={"ID":"4cbd8171-cc73-4d74-84f6-967b722e3f75","Type":"ContainerDied","Data":"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"}
Jul 31 14:50:13.925271 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.925172    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-798f5c847c-rbbwd" event={"ID":"4cbd8171-cc73-4d74-84f6-967b722e3f75","Type":"ContainerDied","Data":"b5aaac31ecbb6abdd6e6043e2c7b682e656cbedd6efbc19ae431d3ca2de15c5c"}
Jul 31 14:50:13.925271 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.925198    2557 scope.go:117] "RemoveContainer" containerID="b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"
Jul 31 14:50:13.933943 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.933924    2557 scope.go:117] "RemoveContainer" containerID="b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"
Jul 31 14:50:13.934299 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:50:13.934274    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46\": container with ID starting with b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46 not found: ID does not exist" containerID="b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"
Jul 31 14:50:13.934350 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.934314    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46"} err="failed to get container status \"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46\": rpc error: code = NotFound desc = could not find container \"b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46\": container with ID starting with b148628200c2ec23f1ea3a3557ed995e0ab21fe97c8271832c7ad9cc7e2b0b46 not found: ID does not exist"
Jul 31 14:50:13.948431 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.948397    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:50:13.951974 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:13.951941    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-798f5c847c-rbbwd"]
Jul 31 14:50:14.247138 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:14.247059    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cbd8171-cc73-4d74-84f6-967b722e3f75" path="/var/lib/kubelet/pods/4cbd8171-cc73-4d74-84f6-967b722e3f75/volumes"
Jul 31 14:50:16.939897 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:16.939867    2557 generic.go:356] "Generic (PLEG): container finished" podID="f0a42421-6f22-42af-99a2-15f971f5c897" containerID="417ca96e2dd831ce779334cdf740e3e905bec4e5a0470b8fbaa738e3dbc7482f" exitCode=0
Jul 31 14:50:16.940381 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:16.939917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wdlx9" event={"ID":"f0a42421-6f22-42af-99a2-15f971f5c897","Type":"ContainerDied","Data":"417ca96e2dd831ce779334cdf740e3e905bec4e5a0470b8fbaa738e3dbc7482f"}
Jul 31 14:50:16.940381 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:16.940230    2557 scope.go:117] "RemoveContainer" containerID="417ca96e2dd831ce779334cdf740e3e905bec4e5a0470b8fbaa738e3dbc7482f"
Jul 31 14:50:17.961866 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:17.961826    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-wdlx9" event={"ID":"f0a42421-6f22-42af-99a2-15f971f5c897","Type":"ContainerStarted","Data":"fef66d3d64a27a7f4f0f6f0a6b3716dba3b4811a9e64834064b266e94656d5c3"}
Jul 31 14:50:30.366935 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.366873    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-564cb897f6-v8gwp" podUID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" containerName="console" containerID="cri-o://38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f" gracePeriod=15
Jul 31 14:50:30.699867 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.699466    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:50:30.767465 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767432    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767465 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767473    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767704 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767524    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767704 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767555    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767704 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767596    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz8mj\" (UniqueName: \"kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767704 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767688    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert\") pod \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\" (UID: \"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3\") "
Jul 31 14:50:30.767912 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767878    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config" (OuterVolumeSpecName: "console-config") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:30.767912 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767895    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:30.768028 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.767998    2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:30.768028 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.768015    2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-service-ca\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:30.768258 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.768210    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:30.769969 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.769940    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:30.770060 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.769982    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:30.770103 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.770055    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj" (OuterVolumeSpecName: "kube-api-access-nz8mj") pod "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" (UID: "8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3"). InnerVolumeSpecName "kube-api-access-nz8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:30.869076 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.868974    2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-oauth-serving-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:30.869076 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.869015    2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-oauth-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:30.869076 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.869030    2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-console-serving-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:30.869076 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:30.869044    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz8mj\" (UniqueName: \"kubernetes.io/projected/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3-kube-api-access-nz8mj\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:50:31.025257 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.021310    2557 generic.go:356] "Generic (PLEG): container finished" podID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" containerID="38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f" exitCode=0
Jul 31 14:50:31.025257 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.021440    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cb897f6-v8gwp" event={"ID":"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3","Type":"ContainerDied","Data":"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"}
Jul 31 14:50:31.025257 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.021496    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-564cb897f6-v8gwp" event={"ID":"8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3","Type":"ContainerDied","Data":"1ae52a69cab6491d8c419c288a65813311407680d4bc35ad47bd632706180e4a"}
Jul 31 14:50:31.025257 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.021527    2557 scope.go:117] "RemoveContainer" containerID="38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"
Jul 31 14:50:31.025257 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.021702    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-564cb897f6-v8gwp"
Jul 31 14:50:31.043220 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.043197    2557 scope.go:117] "RemoveContainer" containerID="38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"
Jul 31 14:50:31.043581 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:50:31.043555    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f\": container with ID starting with 38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f not found: ID does not exist" containerID="38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"
Jul 31 14:50:31.043634 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.043595    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f"} err="failed to get container status \"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f\": rpc error: code = NotFound desc = could not find container \"38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f\": container with ID starting with 38030a5aa510311dc58682d5e59fe5ab3acb9a62dd8eb79dd8154ff51d546f5f not found: ID does not exist"
Jul 31 14:50:31.058139 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.058103    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:50:31.060211 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:31.060187    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-564cb897f6-v8gwp"]
Jul 31 14:50:32.253057 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:32.253011    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" path="/var/lib/kubelet/pods/8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3/volumes"
Jul 31 14:50:47.873387 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:47.873352    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:47.949895 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:47.949853    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:48.108751 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:50:48.108725    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:01.600572 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600531    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:01.601044 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600934    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-metric" containerID="cri-o://cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3" gracePeriod=120
Jul 31 14:51:01.601044 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600949    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-web" containerID="cri-o://2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3" gracePeriod=120
Jul 31 14:51:01.601044 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600994    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="prom-label-proxy" containerID="cri-o://f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f" gracePeriod=120
Jul 31 14:51:01.601220 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600919    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="alertmanager" containerID="cri-o://38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190" gracePeriod=120
Jul 31 14:51:01.601220 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.600973    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy" containerID="cri-o://38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680" gracePeriod=120
Jul 31 14:51:01.601220 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:01.601126    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="config-reloader" containerID="cri-o://b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f" gracePeriod=120
Jul 31 14:51:02.150027 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.149996    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f" exitCode=0
Jul 31 14:51:02.150027 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150020    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3" exitCode=0
Jul 31 14:51:02.150027 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150030    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680" exitCode=0
Jul 31 14:51:02.150027 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150037    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f" exitCode=0
Jul 31 14:51:02.150027 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150042    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190" exitCode=0
Jul 31 14:51:02.150360 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150077    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"}
Jul 31 14:51:02.150360 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150148    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"}
Jul 31 14:51:02.150360 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150174    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"}
Jul 31 14:51:02.150360 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"}
Jul 31 14:51:02.150360 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.150222    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"}
Jul 31 14:51:02.846796 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.846765    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:02.946734 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946644    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.946734 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946686    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.946734 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946719    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946762    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946784    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946821    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946885    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946926    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947010 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.946962    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.947027    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.947059    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t675w\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.947680 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.947083    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca\") pod \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\" (UID: \"9aac8588-6d8b-4de5-b892-3c0a6525d5e2\") "
Jul 31 14:51:02.948091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.947847    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:02.948091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.948061    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:02.948399 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.948378    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:02.950044 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.950014    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.950147 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.950060    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume" (OuterVolumeSpecName: "config-volume") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.950327 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.950300    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.950481 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.950461    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.951411 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.951391    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out" (OuterVolumeSpecName: "config-out") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:02.951491 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.951403    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.951533 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.951489    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w" (OuterVolumeSpecName: "kube-api-access-t675w") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "kube-api-access-t675w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:02.951567 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.951523    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:02.954105 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.954072    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:02.961036 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:02.961014    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config" (OuterVolumeSpecName: "web-config") pod "9aac8588-6d8b-4de5-b892-3c0a6525d5e2" (UID: "9aac8588-6d8b-4de5-b892-3c0a6525d5e2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:03.048217 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048177    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-main-db\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048217 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048210    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t675w\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-kube-api-access-t675w\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048217 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048223    2557 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-metrics-client-ca\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048262    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048278    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-main-tls\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048290    2557 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-volume\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048326    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-web-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048337    2557 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-cluster-tls-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048349    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-tls-assets\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048361    2557 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048373    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-config-out\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048384    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.048507 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.048399    2557 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9aac8588-6d8b-4de5-b892-3c0a6525d5e2-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:03.156247 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.156194    2557 generic.go:356] "Generic (PLEG): container finished" podID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerID="2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3" exitCode=0
Jul 31 14:51:03.156446 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.156273    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"}
Jul 31 14:51:03.156446 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.156291    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.156446 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.156344    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9aac8588-6d8b-4de5-b892-3c0a6525d5e2","Type":"ContainerDied","Data":"a7f403d4e841282cff18acbef402a736952e17d4ee0dd30f9056a0e514fbb9bd"}
Jul 31 14:51:03.156446 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.156379    2557 scope.go:117] "RemoveContainer" containerID="f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"
Jul 31 14:51:03.164625 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.164522    2557 scope.go:117] "RemoveContainer" containerID="cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"
Jul 31 14:51:03.171974 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.171955    2557 scope.go:117] "RemoveContainer" containerID="38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"
Jul 31 14:51:03.179276 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.179257    2557 scope.go:117] "RemoveContainer" containerID="2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"
Jul 31 14:51:03.181051 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.181032    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:03.185769 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.185735    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:03.187582 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.187566    2557 scope.go:117] "RemoveContainer" containerID="b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"
Jul 31 14:51:03.195650 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.195627    2557 scope.go:117] "RemoveContainer" containerID="38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"
Jul 31 14:51:03.203811 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.203787    2557 scope.go:117] "RemoveContainer" containerID="1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb"
Jul 31 14:51:03.211452 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.211430    2557 scope.go:117] "RemoveContainer" containerID="f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"
Jul 31 14:51:03.211930 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.211899    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f\": container with ID starting with f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f not found: ID does not exist" containerID="f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"
Jul 31 14:51:03.212047 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.211961    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f"} err="failed to get container status \"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f\": rpc error: code = NotFound desc = could not find container \"f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f\": container with ID starting with f69f100e7e2599f3fc12373e2685f36e50a5c531e9217e46db9dbb596706c20f not found: ID does not exist"
Jul 31 14:51:03.212047 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.211991    2557 scope.go:117] "RemoveContainer" containerID="cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"
Jul 31 14:51:03.212490 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.212341    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3\": container with ID starting with cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3 not found: ID does not exist" containerID="cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"
Jul 31 14:51:03.212490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.212376    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3"} err="failed to get container status \"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3\": rpc error: code = NotFound desc = could not find container \"cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3\": container with ID starting with cb7ff97193f38717424e6a568018bf64a58bc4045c36084a67c60689b8cfbed3 not found: ID does not exist"
Jul 31 14:51:03.212490 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.212401    2557 scope.go:117] "RemoveContainer" containerID="38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"
Jul 31 14:51:03.213047 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.212988    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680\": container with ID starting with 38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680 not found: ID does not exist" containerID="38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"
Jul 31 14:51:03.213158 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.213066    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680"} err="failed to get container status \"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680\": rpc error: code = NotFound desc = could not find container \"38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680\": container with ID starting with 38f3e1b0d2861638ff5769fa9b65244ea7a357a8a4698801cf5902c62477e680 not found: ID does not exist"
Jul 31 14:51:03.213158 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.213097    2557 scope.go:117] "RemoveContainer" containerID="2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"
Jul 31 14:51:03.213495 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.213473    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3\": container with ID starting with 2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3 not found: ID does not exist" containerID="2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"
Jul 31 14:51:03.213651 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.213626    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3"} err="failed to get container status \"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3\": rpc error: code = NotFound desc = could not find container \"2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3\": container with ID starting with 2916d1d1d10c52a553a118bda0fdf4c33bb9e6b570716844949bf08f1ac7e0c3 not found: ID does not exist"
Jul 31 14:51:03.213770 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.213651    2557 scope.go:117] "RemoveContainer" containerID="b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"
Jul 31 14:51:03.214005 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.213985    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f\": container with ID starting with b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f not found: ID does not exist" containerID="b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"
Jul 31 14:51:03.214084 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214008    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f"} err="failed to get container status \"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f\": rpc error: code = NotFound desc = could not find container \"b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f\": container with ID starting with b23e0ce3a11e63c45d9523cc6f9ab4200bc5c70e32eea8cb3a66e7217432a27f not found: ID does not exist"
Jul 31 14:51:03.214084 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214026    2557 scope.go:117] "RemoveContainer" containerID="38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"
Jul 31 14:51:03.214084 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214030    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:03.214374 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.214354    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190\": container with ID starting with 38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190 not found: ID does not exist" containerID="38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"
Jul 31 14:51:03.214467 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214379    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190"} err="failed to get container status \"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190\": rpc error: code = NotFound desc = could not find container \"38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190\": container with ID starting with 38409ceed089e4e0dea4f1bcdd100dc8d0da810c4cc4b24cc6cc8b1258a50190 not found: ID does not exist"
Jul 31 14:51:03.214467 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214394    2557 scope.go:117] "RemoveContainer" containerID="1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb"
Jul 31 14:51:03.214467 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214448    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="prom-label-proxy"
Jul 31 14:51:03.214467 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214466    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="prom-label-proxy"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214480    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214489    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214500    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-web"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214508    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-web"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214533    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="alertmanager"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214544    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="alertmanager"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214552    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="config-reloader"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214560    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="config-reloader"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214570    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-metric"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214578    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-metric"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214604    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="init-config-reloader"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214613    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="init-config-reloader"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214626    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cbd8171-cc73-4d74-84f6-967b722e3f75" containerName="registry"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214635    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cbd8171-cc73-4d74-84f6-967b722e3f75" containerName="registry"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214651    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" containerName="console"
Jul 31 14:51:03.214694 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214659    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" containerName="console"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214737    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-metric"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214747    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy-web"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214754    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="prom-label-proxy"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214760    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="alertmanager"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214766    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cbd8171-cc73-4d74-84f6-967b722e3f75" containerName="registry"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214771    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="config-reloader"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214777    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5ad9e1-e995-4d0c-a141-9f468e2f6cb3" containerName="console"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:03.214776    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb\": container with ID starting with 1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb not found: ID does not exist" containerID="1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214798    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb"} err="failed to get container status \"1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb\": rpc error: code = NotFound desc = could not find container \"1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb\": container with ID starting with 1d8a6af9efd6284f53a12cf26b7ff335bf1504cb570f1aef6cb22fff6c8837cb not found: ID does not exist"
Jul 31 14:51:03.215163 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.214782    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" containerName="kube-rbac-proxy"
Jul 31 14:51:03.221548 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.221522    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.224070 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224048    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 31 14:51:03.224443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224339    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 31 14:51:03.224443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224351    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tczp9\""
Jul 31 14:51:03.224443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224393    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 31 14:51:03.224443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224400    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 31 14:51:03.224443 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224431    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 31 14:51:03.224754 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224420    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 31 14:51:03.224754 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224472    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 31 14:51:03.224754 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.224434    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 31 14:51:03.229629 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.229587    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 31 14:51:03.231813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.231793    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:03.250380 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250339    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250380 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250384    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-web-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250405    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250489    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250515    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250554    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250596 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250577    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250849 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-config-out\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250849 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250778    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5gbw\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-kube-api-access-h5gbw\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.250910 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.250854    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-config-volume\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351470 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351428    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-config-volume\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351479    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351514    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-web-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351540    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351660    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351747    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351784    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351813    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351859    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.351892 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351887    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.352049 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351950    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-config-out\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.352049 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351960    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.352049 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.351981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5gbw\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-kube-api-access-h5gbw\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.354631 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.354597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.354917 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.354893    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355273 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355369 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355223    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355369 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355310    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355369 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355685 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355659    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355724 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355664    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-config-volume\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355724 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355668    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9975b93a-f021-4c64-a616-c070a21c4886-web-config\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.355893 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.355875    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9975b93a-f021-4c64-a616-c070a21c4886-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.356176 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.356162    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9975b93a-f021-4c64-a616-c070a21c4886-config-out\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.362616 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.362592    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5gbw\" (UniqueName: \"kubernetes.io/projected/9975b93a-f021-4c64-a616-c070a21c4886-kube-api-access-h5gbw\") pod \"alertmanager-main-0\" (UID: \"9975b93a-f021-4c64-a616-c070a21c4886\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.532179 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.532097    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 31 14:51:03.672668 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:03.672561    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 31 14:51:03.675332 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:51:03.675302    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9975b93a_f021_4c64_a616_c070a21c4886.slice/crio-6121dd179e90333d4c1e4c1b5484dae259705aba57fb83bc8d3fa0e600d1b9d2 WatchSource:0}: Error finding container 6121dd179e90333d4c1e4c1b5484dae259705aba57fb83bc8d3fa0e600d1b9d2: Status 404 returned error can't find the container with id 6121dd179e90333d4c1e4c1b5484dae259705aba57fb83bc8d3fa0e600d1b9d2
Jul 31 14:51:04.161957 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:04.161925    2557 generic.go:356] "Generic (PLEG): container finished" podID="9975b93a-f021-4c64-a616-c070a21c4886" containerID="4b4cfefad011d7071945bbb48224a4b38a333afb424431b7bb04ad60a4d277a7" exitCode=0
Jul 31 14:51:04.162390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:04.162015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerDied","Data":"4b4cfefad011d7071945bbb48224a4b38a333afb424431b7bb04ad60a4d277a7"}
Jul 31 14:51:04.162390 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:04.162092    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"6121dd179e90333d4c1e4c1b5484dae259705aba57fb83bc8d3fa0e600d1b9d2"}
Jul 31 14:51:04.247171 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:04.247141    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aac8588-6d8b-4de5-b892-3c0a6525d5e2" path="/var/lib/kubelet/pods/9aac8588-6d8b-4de5-b892-3c0a6525d5e2/volumes"
Jul 31 14:51:05.171060 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171019    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"78d51279d31f3e618a7f89f233c2cd9ab5a92f582eaceee25acea98c0d4d6c28"}
Jul 31 14:51:05.171585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171108    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"00165d74bc8619e681fa377c2e51b39884a4b25cc1b05fbe1bcb899d2de2c4b0"}
Jul 31 14:51:05.171585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171135    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"9463669af38cdb90ee575cff12e180cd3493be7aef8ca60d66cc540d4b82ff8f"}
Jul 31 14:51:05.171585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171165    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"bc26c64942a40007a4aa8b274e7a3f6e3358cbea462e38cdb8e3893673b2ab7d"}
Jul 31 14:51:05.171585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171189    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"e94aa2bd63a7453dfddf0c0d1572a38b8bbe5ffe1621b112ee089fd28112765e"}
Jul 31 14:51:05.171585 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.171212    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9975b93a-f021-4c64-a616-c070a21c4886","Type":"ContainerStarted","Data":"9478c660a230fa1fc09ac0cc410067e35a2a8e67deecf99f645e37880f4dfc72"}
Jul 31 14:51:05.200795 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.200737    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.184217133 podStartE2EDuration="2.200718455s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.501322ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:05.198270333 +0000 UTC m=+187.356521823" watchObservedRunningTime="2026-07-31 14:51:05.200718455 +0000 UTC m=+187.358969944"
Jul 31 14:51:05.618046 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.617967    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-94cb45949-qxdwh"]
Jul 31 14:51:05.626377 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.626349    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.630569 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630542    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 31 14:51:05.630699 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630632    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 31 14:51:05.630772 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630701    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 31 14:51:05.630772 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630732    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 31 14:51:05.630883 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630823    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 31 14:51:05.630945 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.630925    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6ltgt\""
Jul 31 14:51:05.637989 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.637961    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-94cb45949-qxdwh"]
Jul 31 14:51:05.638670 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.638649    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 31 14:51:05.672178 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672145    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672197    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672248    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-serving-certs-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-metrics-client-ca\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqvzv\" (UniqueName: \"kubernetes.io/projected/dbe1f860-18ee-443f-91c5-076bfcc88504-kube-api-access-mqvzv\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672404 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672381    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-federate-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.672644 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.672441    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773050 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773050 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773061    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-serving-certs-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773121    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-metrics-client-ca\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773164    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqvzv\" (UniqueName: \"kubernetes.io/projected/dbe1f860-18ee-443f-91c5-076bfcc88504-kube-api-access-mqvzv\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773202    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-federate-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773294    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.773401 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.773347    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.774765 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.774318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.774765 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.774413    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-serving-certs-ca-bundle\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.774765 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.774583    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe1f860-18ee-443f-91c5-076bfcc88504-metrics-client-ca\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.776756 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.776723    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.777602 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.777573    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-telemeter-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.777769 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.777597    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.777769 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.777614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/dbe1f860-18ee-443f-91c5-076bfcc88504-federate-client-tls\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.785217 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.783832    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqvzv\" (UniqueName: \"kubernetes.io/projected/dbe1f860-18ee-443f-91c5-076bfcc88504-kube-api-access-mqvzv\") pod \"telemeter-client-94cb45949-qxdwh\" (UID: \"dbe1f860-18ee-443f-91c5-076bfcc88504\") " pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.942406 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.942363    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh"
Jul 31 14:51:05.946476 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946442    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946735    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus" containerID="cri-o://4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" gracePeriod=600
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946767    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-web" containerID="cri-o://a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3" gracePeriod=600
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946785    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="config-reloader" containerID="cri-o://58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d" gracePeriod=600
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946887    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="thanos-sidecar" containerID="cri-o://105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be" gracePeriod=600
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946766    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy" containerID="cri-o://7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb" gracePeriod=600
Jul 31 14:51:05.947054 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:05.946886    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2" gracePeriod=600
Jul 31 14:51:06.152399 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.152369    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-94cb45949-qxdwh"]
Jul 31 14:51:06.156253 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:51:06.156214    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe1f860_18ee_443f_91c5_076bfcc88504.slice/crio-840f45726b01fd3f0f4507e990e36d8d7aa7c026e31ddf25ecddc821389457e8 WatchSource:0}: Error finding container 840f45726b01fd3f0f4507e990e36d8d7aa7c026e31ddf25ecddc821389457e8: Status 404 returned error can't find the container with id 840f45726b01fd3f0f4507e990e36d8d7aa7c026e31ddf25ecddc821389457e8
Jul 31 14:51:06.178830 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178805    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3" exitCode=0
Jul 31 14:51:06.178830 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178828    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be" exitCode=0
Jul 31 14:51:06.179226 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178837    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d" exitCode=0
Jul 31 14:51:06.179226 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178848    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"}
Jul 31 14:51:06.179226 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178924    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"}
Jul 31 14:51:06.179226 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.178959    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"}
Jul 31 14:51:06.180172 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:06.180145    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh" event={"ID":"dbe1f860-18ee-443f-91c5-076bfcc88504","Type":"ContainerStarted","Data":"840f45726b01fd3f0f4507e990e36d8d7aa7c026e31ddf25ecddc821389457e8"}
Jul 31 14:51:07.195990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:07.195951    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2" exitCode=0
Jul 31 14:51:07.195990 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:07.195988    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb" exitCode=0
Jul 31 14:51:07.196527 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:07.196042    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"}
Jul 31 14:51:07.196527 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:07.196129    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"}
Jul 31 14:51:08.094950 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:08.094856    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:08.095914 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:08.095873    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:08.096756 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:08.096726    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:08.096842 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:08.096764    2557 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus" probeResult="unknown"
Jul 31 14:51:08.204844 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:08.204807    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh" event={"ID":"dbe1f860-18ee-443f-91c5-076bfcc88504","Type":"ContainerStarted","Data":"34681054f89fd25523e4060a3e3f4d24444cc26d414579495975a4a1eae48c52"}
Jul 31 14:51:08.205312 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:08.204872    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh" event={"ID":"dbe1f860-18ee-443f-91c5-076bfcc88504","Type":"ContainerStarted","Data":"7b513ab147dae010b8b4aec80addfc59e7c60ca303e1422c3c1e72232ffa48b0"}
Jul 31 14:51:08.205312 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:08.204898    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh" event={"ID":"dbe1f860-18ee-443f-91c5-076bfcc88504","Type":"ContainerStarted","Data":"964cbcf17363432f0640323509dd7cc354bd46739e52330e42d8efc1201381d1"}
Jul 31 14:51:08.228093 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:08.228033    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-94cb45949-qxdwh" podStartSLOduration=1.6386505850000002 podStartE2EDuration="3.228013746s" totalImagesPullingTime="1.589363161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:08.226423945 +0000 UTC m=+190.384675436" watchObservedRunningTime="2026-07-31 14:51:08.228013746 +0000 UTC m=+190.386265238"
Jul 31 14:51:13.095149 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:13.095102    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:13.096108 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:13.096073    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:13.097022 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:13.096991    2557 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Jul 31 14:51:13.097132 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:13.097039    2557 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus" probeResult="unknown"
Jul 31 14:51:13.689982 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.689958    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:13.747042 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747012    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747042 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747045    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747100    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747136    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747168    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747221    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747289 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747267    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r2g5\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747300    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747331    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747354    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747383    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747425    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747469    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747495    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747529    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.747588 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747540    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.748013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747598    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.748013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747654    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.748013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747680    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.748013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747797    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8b7cc88e-164a-4506-944d-22675354a3c9\" (UID: \"8b7cc88e-164a-4506-944d-22675354a3c9\") "
Jul 31 14:51:13.748013 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.747853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.748275 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.748074    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.748275 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.748092    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.748275 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.748161    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.749367 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.748513    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.749367 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.748840    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.749528 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.749395    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.751763 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.751417    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:13.751867 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.751793    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:13.753693 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.753667    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.754200 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.754177    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config" (OuterVolumeSpecName: "config") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.754637 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.754560    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:13.754728 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.754657    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.755012 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.754984    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.755114 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755063    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.755303 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755266    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5" (OuterVolumeSpecName: "kube-api-access-5r2g5") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "kube-api-access-5r2g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:13.755536 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.755637 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755580    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.755899 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755867    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out" (OuterVolumeSpecName: "config-out") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:13.756022 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.755998    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.767556 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.767534    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config" (OuterVolumeSpecName: "web-config") pod "8b7cc88e-164a-4506-944d-22675354a3c9" (UID: "8b7cc88e-164a-4506-944d-22675354a3c9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:13.849124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849074    2557 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-grpc-tls\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849118    2557 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-web-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849128    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849137    2557 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849124 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849146    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r2g5\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-kube-api-access-5r2g5\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849155    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849166    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849174    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849184    2557 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-kube-rbac-proxy\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849194    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849202    2557 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-config-out\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849211    2557 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b7cc88e-164a-4506-944d-22675354a3c9-configmap-metrics-client-ca\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849220    2557 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b7cc88e-164a-4506-944d-22675354a3c9-tls-assets\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849229    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849258    2557 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b7cc88e-164a-4506-944d-22675354a3c9-prometheus-k8s-db\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849266    2557 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-metrics-client-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849274    2557 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:13.849468 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:13.849282    2557 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b7cc88e-164a-4506-944d-22675354a3c9-config\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 14:51:14.234372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.234338    2557 generic.go:356] "Generic (PLEG): container finished" podID="8b7cc88e-164a-4506-944d-22675354a3c9" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392" exitCode=0
Jul 31 14:51:14.234778 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.234424    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.234778 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.234427    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"}
Jul 31 14:51:14.234778 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.234497    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b7cc88e-164a-4506-944d-22675354a3c9","Type":"ContainerDied","Data":"188241d38dcf1f1a6229a099254771f3b1c9d6162b776b43a5283ba20b62ac9f"}
Jul 31 14:51:14.234778 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.234533    2557 scope.go:117] "RemoveContainer" containerID="60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"
Jul 31 14:51:14.242781 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.242760    2557 scope.go:117] "RemoveContainer" containerID="7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"
Jul 31 14:51:14.250627 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.250609    2557 scope.go:117] "RemoveContainer" containerID="a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"
Jul 31 14:51:14.257721 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.257700    2557 scope.go:117] "RemoveContainer" containerID="105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"
Jul 31 14:51:14.263056 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.263027    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:14.265787 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.265766    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:14.266100 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.266085    2557 scope.go:117] "RemoveContainer" containerID="58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"
Jul 31 14:51:14.273364 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.273342    2557 scope.go:117] "RemoveContainer" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"
Jul 31 14:51:14.281037 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.281020    2557 scope.go:117] "RemoveContainer" containerID="9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351"
Jul 31 14:51:14.287927 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.287907    2557 scope.go:117] "RemoveContainer" containerID="60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"
Jul 31 14:51:14.288198 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.288173    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2\": container with ID starting with 60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2 not found: ID does not exist" containerID="60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"
Jul 31 14:51:14.288278 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288205    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2"} err="failed to get container status \"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2\": rpc error: code = NotFound desc = could not find container \"60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2\": container with ID starting with 60251b0d056eadfb897dbf6a8cade866345e951a83b151bac813df55242bc1c2 not found: ID does not exist"
Jul 31 14:51:14.288278 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288224    2557 scope.go:117] "RemoveContainer" containerID="7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"
Jul 31 14:51:14.288472 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.288450    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb\": container with ID starting with 7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb not found: ID does not exist" containerID="7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"
Jul 31 14:51:14.288526 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288477    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb"} err="failed to get container status \"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb\": rpc error: code = NotFound desc = could not find container \"7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb\": container with ID starting with 7377a671abb39e06dc22d3e36a548fb60fb85a3c56529999bf44513ed0171adb not found: ID does not exist"
Jul 31 14:51:14.288526 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288491    2557 scope.go:117] "RemoveContainer" containerID="a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"
Jul 31 14:51:14.288747 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.288727    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3\": container with ID starting with a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3 not found: ID does not exist" containerID="a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"
Jul 31 14:51:14.288782 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288753    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3"} err="failed to get container status \"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3\": rpc error: code = NotFound desc = could not find container \"a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3\": container with ID starting with a9737c69e760f32a9f824c284cd186421e3b04187974f19b7e09dec8676e18e3 not found: ID does not exist"
Jul 31 14:51:14.288782 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.288771    2557 scope.go:117] "RemoveContainer" containerID="105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"
Jul 31 14:51:14.288998 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.288980    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be\": container with ID starting with 105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be not found: ID does not exist" containerID="105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"
Jul 31 14:51:14.289102 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289002    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be"} err="failed to get container status \"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be\": rpc error: code = NotFound desc = could not find container \"105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be\": container with ID starting with 105e096affaf42dbbe2825926aeae418233ee6fcd66abd9dad3bbeaf910d97be not found: ID does not exist"
Jul 31 14:51:14.289102 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289018    2557 scope.go:117] "RemoveContainer" containerID="58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"
Jul 31 14:51:14.289279 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.289256    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d\": container with ID starting with 58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d not found: ID does not exist" containerID="58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"
Jul 31 14:51:14.289329 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289287    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d"} err="failed to get container status \"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d\": rpc error: code = NotFound desc = could not find container \"58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d\": container with ID starting with 58edbe8b933e4302196051a2cc9089aa19034ce4062850a93e7cc9a80872250d not found: ID does not exist"
Jul 31 14:51:14.289329 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289303    2557 scope.go:117] "RemoveContainer" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"
Jul 31 14:51:14.289558 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.289541    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392\": container with ID starting with 4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392 not found: ID does not exist" containerID="4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"
Jul 31 14:51:14.289597 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289561    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392"} err="failed to get container status \"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392\": rpc error: code = NotFound desc = could not find container \"4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392\": container with ID starting with 4a5f047705e9405bc282bf6c73bda847f1f3f44009c8ce1da7be91f5a73c9392 not found: ID does not exist"
Jul 31 14:51:14.289597 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289576    2557 scope.go:117] "RemoveContainer" containerID="9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351"
Jul 31 14:51:14.289797 ip-10-0-137-96 kubenswrapper[2557]: E0731 14:51:14.289782    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351\": container with ID starting with 9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351 not found: ID does not exist" containerID="9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351"
Jul 31 14:51:14.289832 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.289799    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351"} err="failed to get container status \"9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351\": rpc error: code = NotFound desc = could not find container \"9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351\": container with ID starting with 9ffa87ecc48f21789823ee6c115f37c59e390fe533c9bf714b97e9fdfc880351 not found: ID does not exist"
Jul 31 14:51:14.305212 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305177    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:14.305893 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305873    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-web"
Jul 31 14:51:14.305934 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305897    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-web"
Jul 31 14:51:14.305934 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305924    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="config-reloader"
Jul 31 14:51:14.306000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305934    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="config-reloader"
Jul 31 14:51:14.306000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305952    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="thanos-sidecar"
Jul 31 14:51:14.306000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305961    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="thanos-sidecar"
Jul 31 14:51:14.306000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305975    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="init-config-reloader"
Jul 31 14:51:14.306000 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.305984    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="init-config-reloader"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306002    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306011    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306022    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306037    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306067    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:14.306168 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306076    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306206    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="thanos-sidecar"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306218    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306227    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="config-reloader"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306269    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306279    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="kube-rbac-proxy-web"
Jul 31 14:51:14.306388 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.306293    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" containerName="prometheus"
Jul 31 14:51:14.313534 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.313512    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.316334 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.316316    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 14:51:14.320574 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320549    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 14:51:14.320574 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320560    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 14:51:14.320755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320583    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 14:51:14.320755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320676    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 14:51:14.320755 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320683    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 14:51:14.320971 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320889    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 14:51:14.320971 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320916    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 14:51:14.320971 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320948    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 14:51:14.321159 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320947    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4n5funqrsc9d3\""
Jul 31 14:51:14.321159 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320949    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 14:51:14.321159 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.320952    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6q4cb\""
Jul 31 14:51:14.321159 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.321084    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 14:51:14.321159 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.321153    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 14:51:14.321489 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.321473    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 14:51:14.324845 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.324822    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 14:51:14.327762 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.327747    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 14:51:14.334660 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.334639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:14.453814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453781    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.453814 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453847    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453867    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453928    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.453990    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454020    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454068    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tckh4\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-kube-api-access-tckh4\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454092    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454156    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454180 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454178    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454196    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454266    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454322    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454372    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.454410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.454393    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555112 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555076    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555112 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555114    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555357 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555142    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555357 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555279    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555357 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555502 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555362    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555502 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555400    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555502 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555424    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555502 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555502 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555510    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555559    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tckh4\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-kube-api-access-tckh4\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555595    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555658    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.555709 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555734    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555795    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555809    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.555955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556308 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.556080    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.556371 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.556313    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.557887 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.556938    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.557887 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.557140    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.559195 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.559176    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.559841 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.559814    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.559930 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.559870    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560158 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560135    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560382 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560353    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560538 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560632 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560596    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-config\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560687 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560665    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560737 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560688    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560951 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560930    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.560951 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.560938    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.561811 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.561791    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1c424a8-c7fe-48dd-8688-f0fb80d41479-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.562347 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.562329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1c424a8-c7fe-48dd-8688-f0fb80d41479-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.565618 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.565595    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tckh4\" (UniqueName: \"kubernetes.io/projected/c1c424a8-c7fe-48dd-8688-f0fb80d41479-kube-api-access-tckh4\") pod \"prometheus-k8s-0\" (UID: \"c1c424a8-c7fe-48dd-8688-f0fb80d41479\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.623046 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.623005    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:14.770739 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:14.770666    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:14.774460 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:51:14.774433    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c424a8_c7fe_48dd_8688_f0fb80d41479.slice/crio-d856d5a376a7f3c6af18a72cec1c40356e552968265f90d7b901fea2d121e415 WatchSource:0}: Error finding container d856d5a376a7f3c6af18a72cec1c40356e552968265f90d7b901fea2d121e415: Status 404 returned error can't find the container with id d856d5a376a7f3c6af18a72cec1c40356e552968265f90d7b901fea2d121e415
Jul 31 14:51:15.240372 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:15.240333    2557 generic.go:356] "Generic (PLEG): container finished" podID="c1c424a8-c7fe-48dd-8688-f0fb80d41479" containerID="aeee68594ad5461340ff66ed3ff8831f44d0667e16272d6d023d43b2cd7f376f" exitCode=0
Jul 31 14:51:15.240802 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:15.240425    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerDied","Data":"aeee68594ad5461340ff66ed3ff8831f44d0667e16272d6d023d43b2cd7f376f"}
Jul 31 14:51:15.240802 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:15.240498    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"d856d5a376a7f3c6af18a72cec1c40356e552968265f90d7b901fea2d121e415"}
Jul 31 14:51:16.249669 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.249622    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b7cc88e-164a-4506-944d-22675354a3c9" path="/var/lib/kubelet/pods/8b7cc88e-164a-4506-944d-22675354a3c9/volumes"
Jul 31 14:51:16.254368 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254338    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"32796da69542b001f000273ead7e6c62b4827a4b22b17fbb8fed5e79affa3e24"}
Jul 31 14:51:16.254509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254400    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"404603791cb6cb56d3ffde1c4a9a133b3a9096cabfe5d53c0345aa2ce91cfe02"}
Jul 31 14:51:16.254509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254438    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"b55aa9e28bd7fe85b296b9934b619f9904a86d4fbd3bcaf54c996f24a8da0046"}
Jul 31 14:51:16.254509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254462    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"5882241c73937726fd472061de5f5decb37e389d4a649b42b3e535b6c71c8d86"}
Jul 31 14:51:16.254509 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254501    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"212648afb416f7f67a2b88227779e2c801f61588c2d8b7e6045dff650fc0ff92"}
Jul 31 14:51:16.254636 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.254531    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1c424a8-c7fe-48dd-8688-f0fb80d41479","Type":"ContainerStarted","Data":"73baf490dbfe185254444eabc3286379f07ba1f8e11b974c8759b2866175400d"}
Jul 31 14:51:16.294949 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:16.294887    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.272801146 podStartE2EDuration="2.29486993s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.068784ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:16.294087905 +0000 UTC m=+198.452339408" watchObservedRunningTime="2026-07-31 14:51:16.29486993 +0000 UTC m=+198.453121419"
Jul 31 14:51:19.623368 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:51:19.623332    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:14.624227 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:14.624183    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:14.639889 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:14.639863    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:15.500139 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:15.500112    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:58.140307 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:58.140278    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:52:58.142028 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:58.142005    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:52:58.150873 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:52:58.150851    2557 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:55:04.619185 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.619144    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zzz7j"]
Jul 31 14:55:04.622520 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.622499    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.627968 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.627066    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qt2g8\""
Jul 31 14:55:04.628354 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.628009    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 14:55:04.628354 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.628313    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 14:55:04.643561 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.643536    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zzz7j"]
Jul 31 14:55:04.771703 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.771661    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrzq\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-kube-api-access-knrzq\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.771703 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.771708    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.872548 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.872466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrzq\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-kube-api-access-knrzq\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.872548 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.872519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.880813 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.880787    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.887526 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.887503    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrzq\" (UniqueName: \"kubernetes.io/projected/2f81e631-7a2e-4e3b-876e-48d74258f2f1-kube-api-access-knrzq\") pod \"cert-manager-webhook-6486ffd997-zzz7j\" (UID: \"2f81e631-7a2e-4e3b-876e-48d74258f2f1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:04.944532 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:04.944498    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:05.090056 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:05.090038    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-zzz7j"]
Jul 31 14:55:05.092252 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:55:05.092218    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f81e631_7a2e_4e3b_876e_48d74258f2f1.slice/crio-7a042bc6beaeb4c55adee13e2d72768775b8d2c5817e7c1dca62bb4e0a2f6c87 WatchSource:0}: Error finding container 7a042bc6beaeb4c55adee13e2d72768775b8d2c5817e7c1dca62bb4e0a2f6c87: Status 404 returned error can't find the container with id 7a042bc6beaeb4c55adee13e2d72768775b8d2c5817e7c1dca62bb4e0a2f6c87
Jul 31 14:55:05.094169 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:05.094154    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:55:05.139560 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:05.139474    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j" event={"ID":"2f81e631-7a2e-4e3b-876e-48d74258f2f1","Type":"ContainerStarted","Data":"7a042bc6beaeb4c55adee13e2d72768775b8d2c5817e7c1dca62bb4e0a2f6c87"}
Jul 31 14:55:06.941326 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:06.941290    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9r455"]
Jul 31 14:55:06.945808 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:06.945789    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:06.950025 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:06.950005    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6m96s\""
Jul 31 14:55:06.962662 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:06.962639    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9r455"]
Jul 31 14:55:07.090750 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.090714    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skb6\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-kube-api-access-9skb6\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.090916 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.090784    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.192060 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.191969    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.192060 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.192050    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skb6\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-kube-api-access-9skb6\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.206950 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.206922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.207091 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.207072    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skb6\" (UniqueName: \"kubernetes.io/projected/12702c70-98b0-4cbd-89a7-071d62852f7c-kube-api-access-9skb6\") pod \"cert-manager-cainjector-559cdd8c87-9r455\" (UID: \"12702c70-98b0-4cbd-89a7-071d62852f7c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:07.257167 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:07.257134    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455"
Jul 31 14:55:08.238942 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:08.238920    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9r455"]
Jul 31 14:55:08.241185 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:55:08.241155    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12702c70_98b0_4cbd_89a7_071d62852f7c.slice/crio-4aa307226a2549d5dd129453687e614755cbb709a5e6742c0682057bd0f602cc WatchSource:0}: Error finding container 4aa307226a2549d5dd129453687e614755cbb709a5e6742c0682057bd0f602cc: Status 404 returned error can't find the container with id 4aa307226a2549d5dd129453687e614755cbb709a5e6742c0682057bd0f602cc
Jul 31 14:55:09.161313 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.161278    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j" event={"ID":"2f81e631-7a2e-4e3b-876e-48d74258f2f1","Type":"ContainerStarted","Data":"76809aa2cf0c7592e2e66100d83b22575ff1597d2dc01f2b4c6f76e2457b5955"}
Jul 31 14:55:09.161598 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.161391    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:09.162753 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.162731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455" event={"ID":"12702c70-98b0-4cbd-89a7-071d62852f7c","Type":"ContainerStarted","Data":"0d50e06c2afb461585730f2ada34d7374ad4f1ae1623ce9f35077d76a1b9875d"}
Jul 31 14:55:09.162871 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.162760    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455" event={"ID":"12702c70-98b0-4cbd-89a7-071d62852f7c","Type":"ContainerStarted","Data":"4aa307226a2549d5dd129453687e614755cbb709a5e6742c0682057bd0f602cc"}
Jul 31 14:55:09.203615 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.203572    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j" podStartSLOduration=2.158674687 podStartE2EDuration="5.203558193s" totalImagesPullingTime="3.044883506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:09.202395246 +0000 UTC m=+431.360646736" watchObservedRunningTime="2026-07-31 14:55:09.203558193 +0000 UTC m=+431.361809682"
Jul 31 14:55:09.239682 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:09.239635    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9r455" podStartSLOduration=3.239619349 podStartE2EDuration="3.239619349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:09.238318525 +0000 UTC m=+431.396570015" watchObservedRunningTime="2026-07-31 14:55:09.239619349 +0000 UTC m=+431.397870838"
Jul 31 14:55:15.168808 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:15.168773    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-zzz7j"
Jul 31 14:55:25.002663 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.002580    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s4pxd"]
Jul 31 14:55:25.005801 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.005786    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.009309 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.009287    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xlf2k\""
Jul 31 14:55:25.015006 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.014982    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s4pxd"]
Jul 31 14:55:25.144897 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.144860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.145078 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.144922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p6d\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-kube-api-access-w4p6d\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.246126 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.246093    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p6d\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-kube-api-access-w4p6d\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.246347 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.246155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.256017 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.255947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-bound-sa-token\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.256199 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.256173    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p6d\" (UniqueName: \"kubernetes.io/projected/9082c1f4-76f9-4eff-a4bc-9b74c2be36f6-kube-api-access-w4p6d\") pod \"cert-manager-55b4d954d5-s4pxd\" (UID: \"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6\") " pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.314996 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.314966    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s4pxd"
Jul 31 14:55:25.449716 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:25.449684    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s4pxd"]
Jul 31 14:55:25.453623 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:55:25.453592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9082c1f4_76f9_4eff_a4bc_9b74c2be36f6.slice/crio-c764d5e650f1ba8e567eaedd1b0751c9c74a1e22181923a00e055bb124ea4854 WatchSource:0}: Error finding container c764d5e650f1ba8e567eaedd1b0751c9c74a1e22181923a00e055bb124ea4854: Status 404 returned error can't find the container with id c764d5e650f1ba8e567eaedd1b0751c9c74a1e22181923a00e055bb124ea4854
Jul 31 14:55:26.230783 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:26.230749    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s4pxd" event={"ID":"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6","Type":"ContainerStarted","Data":"46fdd7980c76be15113f3202c6957d024a1be61f3c7164b9d1e147e5032000ce"}
Jul 31 14:55:26.230783 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:26.230787    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s4pxd" event={"ID":"9082c1f4-76f9-4eff-a4bc-9b74c2be36f6","Type":"ContainerStarted","Data":"c764d5e650f1ba8e567eaedd1b0751c9c74a1e22181923a00e055bb124ea4854"}
Jul 31 14:55:26.248040 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:26.247994    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s4pxd" podStartSLOduration=2.247978228 podStartE2EDuration="2.247978228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:55:26.2473296 +0000 UTC m=+448.405581089" watchObservedRunningTime="2026-07-31 14:55:26.247978228 +0000 UTC m=+448.406229718"
Jul 31 14:55:56.632204 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.632168    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"]
Jul 31 14:55:56.635503 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.635486    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.639209 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639183    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 31 14:55:56.639357 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639186    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4hjz4\""
Jul 31 14:55:56.639425 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639386    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:55:56.639485 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639424    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 31 14:55:56.639485 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639440    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 14:55:56.639654 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.639638    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 31 14:55:56.652790 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.652763    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"]
Jul 31 14:55:56.715156 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.715115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjv8\" (UniqueName: \"kubernetes.io/projected/f7c84927-c6ea-46d7-8390-746b208cf3c8-kube-api-access-hwjv8\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.715156 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.715165    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.715402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.715186    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-metrics-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.715402 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.715299    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c84927-c6ea-46d7-8390-746b208cf3c8-manager-config\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.816326 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.816286    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjv8\" (UniqueName: \"kubernetes.io/projected/f7c84927-c6ea-46d7-8390-746b208cf3c8-kube-api-access-hwjv8\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.816499 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.816345    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.816499 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.816365    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-metrics-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.816499 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.816429    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c84927-c6ea-46d7-8390-746b208cf3c8-manager-config\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.817161 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.817138    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7c84927-c6ea-46d7-8390-746b208cf3c8-manager-config\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.818975 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.818947    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-metrics-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.819079 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.819058    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7c84927-c6ea-46d7-8390-746b208cf3c8-cert\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.826275 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.826248    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjv8\" (UniqueName: \"kubernetes.io/projected/f7c84927-c6ea-46d7-8390-746b208cf3c8-kube-api-access-hwjv8\") pod \"lws-controller-manager-56854c75c8-hmj5l\" (UID: \"f7c84927-c6ea-46d7-8390-746b208cf3c8\") " pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:56.945038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:56.945000    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:55:57.122676 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:57.122644    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"]
Jul 31 14:55:57.125924 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:55:57.125897    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c84927_c6ea_46d7_8390_746b208cf3c8.slice/crio-8cc50602086a921f244eabe8e420ee2d46c4f0818c3a15b3d0a59c47cfec4a50 WatchSource:0}: Error finding container 8cc50602086a921f244eabe8e420ee2d46c4f0818c3a15b3d0a59c47cfec4a50: Status 404 returned error can't find the container with id 8cc50602086a921f244eabe8e420ee2d46c4f0818c3a15b3d0a59c47cfec4a50
Jul 31 14:55:57.354157 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:55:57.354071    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l" event={"ID":"f7c84927-c6ea-46d7-8390-746b208cf3c8","Type":"ContainerStarted","Data":"8cc50602086a921f244eabe8e420ee2d46c4f0818c3a15b3d0a59c47cfec4a50"}
Jul 31 14:56:00.369261 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:00.369209    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l" event={"ID":"f7c84927-c6ea-46d7-8390-746b208cf3c8","Type":"ContainerStarted","Data":"d0d6a825e784aa985e1ea889769e3fe0edc8a9d618473a6ed98e1effcf6d4433"}
Jul 31 14:56:00.369700 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:00.369312    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:56:00.391714 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:00.391661    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l" podStartSLOduration=1.833918374 podStartE2EDuration="4.39164543s" totalImagesPullingTime="2.557727056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:00.391526529 +0000 UTC m=+482.549778017" watchObservedRunningTime="2026-07-31 14:56:00.39164543 +0000 UTC m=+482.549896897"
Jul 31 14:56:11.376038 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:11.376009    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56854c75c8-hmj5l"
Jul 31 14:56:18.014512 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.014483    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"]
Jul 31 14:56:18.018046 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.018029    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.021901 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.021870    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 14:56:18.022042 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.022025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q7gd6\""
Jul 31 14:56:18.022139 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.022125    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 14:56:18.034949 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.034923    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"]
Jul 31 14:56:18.101967 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.101932    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhc7\" (UniqueName: \"kubernetes.io/projected/d1a46c41-8870-4367-8be0-09a75a1a29df-kube-api-access-dvhc7\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.102150 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.101992    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a46c41-8870-4367-8be0-09a75a1a29df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.203625 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.203580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a46c41-8870-4367-8be0-09a75a1a29df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.203820 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.203697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhc7\" (UniqueName: \"kubernetes.io/projected/d1a46c41-8870-4367-8be0-09a75a1a29df-kube-api-access-dvhc7\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.204018 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.203994    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1a46c41-8870-4367-8be0-09a75a1a29df-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.217053 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.217017    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhc7\" (UniqueName: \"kubernetes.io/projected/d1a46c41-8870-4367-8be0-09a75a1a29df-kube-api-access-dvhc7\") pod \"kuadrant-operator-controller-manager-66866f5fcc-lnx9l\" (UID: \"d1a46c41-8870-4367-8be0-09a75a1a29df\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.327158 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.327081    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:18.455802 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:18.455779    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"]
Jul 31 14:56:18.458340 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:56:18.458314    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a46c41_8870_4367_8be0_09a75a1a29df.slice/crio-b917c4c8f819d3f186629b8be10c08e5711dad9580394932f5aa6f89a27b37d9 WatchSource:0}: Error finding container b917c4c8f819d3f186629b8be10c08e5711dad9580394932f5aa6f89a27b37d9: Status 404 returned error can't find the container with id b917c4c8f819d3f186629b8be10c08e5711dad9580394932f5aa6f89a27b37d9
Jul 31 14:56:19.448041 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:19.448001    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l" event={"ID":"d1a46c41-8870-4367-8be0-09a75a1a29df","Type":"ContainerStarted","Data":"b917c4c8f819d3f186629b8be10c08e5711dad9580394932f5aa6f89a27b37d9"}
Jul 31 14:56:23.471418 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:23.471377    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l" event={"ID":"d1a46c41-8870-4367-8be0-09a75a1a29df","Type":"ContainerStarted","Data":"ed4eea40a8447e31a3ff9ee1f6625420a53416270c3be70b431c5dd9414a5b56"}
Jul 31 14:56:23.471828 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:23.471445    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:56:23.493921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:23.493873    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l" podStartSLOduration=2.362855413 podStartE2EDuration="6.493856734s" totalImagesPullingTime="4.131001321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:23.491914997 +0000 UTC m=+505.650166486" watchObservedRunningTime="2026-07-31 14:56:23.493856734 +0000 UTC m=+505.652108223"
Jul 31 14:56:34.477921 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:56:34.477890    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-lnx9l"
Jul 31 14:57:06.994577 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:06.994485    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"]
Jul 31 14:57:07.001362 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.001337    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.005554 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.005532    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ffvtl\""
Jul 31 14:57:07.005811 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.005797    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 31 14:57:07.009445 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.009422    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"]
Jul 31 14:57:07.042081 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.042046    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"]
Jul 31 14:57:07.140117 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.140083    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc93181f-bdf0-49eb-bf34-621bb5309908-config-file\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.140334 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.140122    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26blk\" (UniqueName: \"kubernetes.io/projected/bc93181f-bdf0-49eb-bf34-621bb5309908-kube-api-access-26blk\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.241214 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.241178    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc93181f-bdf0-49eb-bf34-621bb5309908-config-file\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.241434 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.241223    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26blk\" (UniqueName: \"kubernetes.io/projected/bc93181f-bdf0-49eb-bf34-621bb5309908-kube-api-access-26blk\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.241908 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.241880    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc93181f-bdf0-49eb-bf34-621bb5309908-config-file\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.250512 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.250446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26blk\" (UniqueName: \"kubernetes.io/projected/bc93181f-bdf0-49eb-bf34-621bb5309908-kube-api-access-26blk\") pod \"limitador-limitador-85d9db4d5d-8q6sk\" (UID: \"bc93181f-bdf0-49eb-bf34-621bb5309908\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.313410 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.313368    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:07.445966 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.445777    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"]
Jul 31 14:57:07.448937 ip-10-0-137-96 kubenswrapper[2557]: W0731 14:57:07.448894    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc93181f_bdf0_49eb_bf34_621bb5309908.slice/crio-559551100b8a889905648623eaf6add6a08f6779d178c0d5ab02f5db61665a41 WatchSource:0}: Error finding container 559551100b8a889905648623eaf6add6a08f6779d178c0d5ab02f5db61665a41: Status 404 returned error can't find the container with id 559551100b8a889905648623eaf6add6a08f6779d178c0d5ab02f5db61665a41
Jul 31 14:57:07.666734 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:07.666654    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk" event={"ID":"bc93181f-bdf0-49eb-bf34-621bb5309908","Type":"ContainerStarted","Data":"559551100b8a889905648623eaf6add6a08f6779d178c0d5ab02f5db61665a41"}
Jul 31 14:57:09.679333 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:09.679296    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk" event={"ID":"bc93181f-bdf0-49eb-bf34-621bb5309908","Type":"ContainerStarted","Data":"4b8b7898d1a0bbc754ed8751995d018846fdf80f241a7fb9edf04dd8a040bcc4"}
Jul 31 14:57:09.679751 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:09.679394    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:20.685399 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:20.685365    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk"
Jul 31 14:57:20.706102 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:20.706051    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-8q6sk" podStartSLOduration=13.326184425 podStartE2EDuration="14.706033975s" totalImagesPullingTime="1.37984955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:09.704472952 +0000 UTC m=+551.862724440" watchObservedRunningTime="2026-07-31 14:57:20.706033975 +0000 UTC m=+562.864285464"
Jul 31 14:57:58.171072 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:58.171046    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 14:57:58.171623 ip-10-0-137-96 kubenswrapper[2557]: I0731 14:57:58.171207    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:01:28.580951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.580877    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:01:28.584377 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.584355    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.586816 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.586795    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 31 15:01:28.586933 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.586859    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-tt4kb\""
Jul 31 15:01:28.587170 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.587153    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 31 15:01:28.587267 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.587224    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 31 15:01:28.587812 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.587790    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:01:28.587812 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.587800    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-czjml\""
Jul 31 15:01:28.587812 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.587794    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 31 15:01:28.598706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.598684    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:01:28.629904 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.629867    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.629904 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.629912    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630137 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.629935    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7mb\" (UniqueName: \"kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630137 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.630001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630137 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.630039    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630137 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.630067    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.630148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.630314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.630182    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731620 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731580    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731620 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731624    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731656    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731685    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731736    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731766    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731797    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7mb\" (UniqueName: \"kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.731860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731836    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.732168 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.731997    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.732168 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.732150    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.732291 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.732191    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.732291 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.732211    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.732466 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.732446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.733909 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.733890    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.734115 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.734096    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.740765 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.740739    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7mb\" (UniqueName: \"kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:28.895100 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:28.895007    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:01:29.029474 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:29.029449    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:01:29.035737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:29.035716    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:01:29.842572 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:29.842524    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerStarted","Data":"7e17de1aac2d2dae7423ebaf930a2bbe9abb57304e9531f335cd5cbbcfc7f736"}
Jul 31 15:01:34.869678 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:34.869638    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerStarted","Data":"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"}
Jul 31 15:01:36.881282 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:36.881221    2557 generic.go:356] "Generic (PLEG): container finished" podID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerID="bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b" exitCode=0
Jul 31 15:01:36.881759 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:01:36.881302    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerDied","Data":"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"}
Jul 31 15:02:05.045040 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:05.044935    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerStarted","Data":"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"}
Jul 31 15:02:05.081682 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:05.081627    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=1.5872404100000002 podStartE2EDuration="37.081609593s" totalImagesPullingTime="33.247617009s" totalInitContainerRuntime="2.246752174s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:05.080865988 +0000 UTC m=+847.239117490" watchObservedRunningTime="2026-07-31 15:02:05.081609593 +0000 UTC m=+847.239861097"
Jul 31 15:02:05.895714 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:05.895668    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:02:08.895995 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:08.895962    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:02:08.897405 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:08.897374    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:02:18.895731 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:18.895686    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:02:28.896094 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:28.896039    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:02:38.896157 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:38.896112    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:02:48.895430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:48.895390    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:02:58.200253 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:58.200160    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:02:58.200688 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:58.200665    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:02:58.896210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:02:58.896156    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:03:08.896391 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:08.896343    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:03:18.896443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:18.896395    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:03:28.896458 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:28.896411    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:03:38.896476 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:38.896420    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": dial tcp 10.132.0.31:8000: connect: connection refused"
Jul 31 15:03:48.905873 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:48.905838    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:03:48.913766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:03:48.913737    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:04:10.167223 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:10.167189    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:04:25.191384 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:25.191343    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" containerID="cri-o://b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f" gracePeriod=45
Jul 31 15:04:25.942756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:25.942726    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:04:26.009185 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009097    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009185 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009144    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009185 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009182    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q7mb\" (UniqueName: \"kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009207    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009340    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.009499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.009364    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm\") pod \"62dff1fa-188b-4f0c-b858-412f26ff18a9\" (UID: \"62dff1fa-188b-4f0c-b858-412f26ff18a9\") "
Jul 31 15:04:26.010120 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.010088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:26.010395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.010348    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:26.010471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.010405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home" (OuterVolumeSpecName: "home") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:26.015297 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.012920    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb" (OuterVolumeSpecName: "kube-api-access-9q7mb") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "kube-api-access-9q7mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:26.015297 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.013003    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm" (OuterVolumeSpecName: "dshm") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:26.020595 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.020486    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:26.027067 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.027030    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:26.035045 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.035012    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62dff1fa-188b-4f0c-b858-412f26ff18a9" (UID: "62dff1fa-188b-4f0c-b858-412f26ff18a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:04:26.111422 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111381    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62dff1fa-188b-4f0c-b858-412f26ff18a9-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111422 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111414    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62dff1fa-188b-4f0c-b858-412f26ff18a9-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111422 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111425    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111434    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111443    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111451    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111458    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7mb\" (UniqueName: \"kubernetes.io/projected/62dff1fa-188b-4f0c-b858-412f26ff18a9-kube-api-access-9q7mb\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.111662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.111466    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62dff1fa-188b-4f0c-b858-412f26ff18a9-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:04:26.746404 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.746365    2557 generic.go:356] "Generic (PLEG): container finished" podID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerID="b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f" exitCode=0
Jul 31 15:04:26.746815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.746441    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 31 15:04:26.746815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.746446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerDied","Data":"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"}
Jul 31 15:04:26.746815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.746502    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"62dff1fa-188b-4f0c-b858-412f26ff18a9","Type":"ContainerDied","Data":"7e17de1aac2d2dae7423ebaf930a2bbe9abb57304e9531f335cd5cbbcfc7f736"}
Jul 31 15:04:26.746815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.746524    2557 scope.go:117] "RemoveContainer" containerID="b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"
Jul 31 15:04:26.755021 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.754999    2557 scope.go:117] "RemoveContainer" containerID="bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"
Jul 31 15:04:26.763431 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.763379    2557 scope.go:117] "RemoveContainer" containerID="b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"
Jul 31 15:04:26.763767 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:04:26.763746    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f\": container with ID starting with b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f not found: ID does not exist" containerID="b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"
Jul 31 15:04:26.763838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.763777    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f"} err="failed to get container status \"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f\": rpc error: code = NotFound desc = could not find container \"b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f\": container with ID starting with b4030ea109e4c7732352bf720394c0b87b6c71d8dd13b4399e4107f6a48d4d0f not found: ID does not exist"
Jul 31 15:04:26.763838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.763796    2557 scope.go:117] "RemoveContainer" containerID="bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"
Jul 31 15:04:26.764082 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:04:26.764062    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b\": container with ID starting with bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b not found: ID does not exist" containerID="bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"
Jul 31 15:04:26.764132 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.764088    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b"} err="failed to get container status \"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b\": rpc error: code = NotFound desc = could not find container \"bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b\": container with ID starting with bd09036140c4108d839c751c0c7262ce6b5eba7500a908ab0c462c6b2459f00b not found: ID does not exist"
Jul 31 15:04:26.765014 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.764995    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:04:26.769362 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.769339    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 31 15:04:26.907660 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:26.907613    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.31:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:04:28.248718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:28.248675    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" path="/var/lib/kubelet/pods/62dff1fa-188b-4f0c-b858-412f26ff18a9/volumes"
Jul 31 15:04:56.310273 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310223    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:04:56.312644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310605    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main"
Jul 31 15:04:56.312644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310617    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main"
Jul 31 15:04:56.312644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310633    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="storage-initializer"
Jul 31 15:04:56.312644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310638    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="storage-initializer"
Jul 31 15:04:56.312644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.310712    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="62dff1fa-188b-4f0c-b858-412f26ff18a9" containerName="main"
Jul 31 15:04:56.313551 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.313535    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.316310 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.316287    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 31 15:04:56.317292 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.317277    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 31 15:04:56.317292 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.317282    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 31 15:04:56.317417 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.317306    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-ss9mt\""
Jul 31 15:04:56.324981 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.324955    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:04:56.480913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.480875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.481102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.480921    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.481102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.480947    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.481102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.481022    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.481102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.481056    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.481277 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.481098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdqsb\" (UniqueName: \"kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582274 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582165    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582274 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582214    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582302    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582333    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582399    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdqsb\" (UniqueName: \"kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582449    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582689 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582644    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.582934 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582907    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.583059 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.582982    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.584854 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.584829    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.585153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.585134    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.593102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.593068    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdqsb\" (UniqueName: \"kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb\") pod \"scheduler-inline-config-test-kserve-8655bc757-6bg5q\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.624429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.624391    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:04:56.772274 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.772248    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:04:56.774962 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:04:56.774928    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e8aba9_ffa4_4990_be9b_411444c54c5e.slice/crio-6f644da84ff112574a901623e0fab669a075633f311f623e1b7420e576aa3b3e WatchSource:0}: Error finding container 6f644da84ff112574a901623e0fab669a075633f311f623e1b7420e576aa3b3e: Status 404 returned error can't find the container with id 6f644da84ff112574a901623e0fab669a075633f311f623e1b7420e576aa3b3e
Jul 31 15:04:56.886040 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:56.885956    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" event={"ID":"39e8aba9-ffa4-4990-be9b-411444c54c5e","Type":"ContainerStarted","Data":"6f644da84ff112574a901623e0fab669a075633f311f623e1b7420e576aa3b3e"}
Jul 31 15:04:58.896951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:58.896916    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" event={"ID":"39e8aba9-ffa4-4990-be9b-411444c54c5e","Type":"ContainerStarted","Data":"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"}
Jul 31 15:04:58.925572 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:58.925509    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" podStartSLOduration=1.350647242 podStartE2EDuration="2.925487197s" totalImagesPullingTime="1.574839955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:58.92233274 +0000 UTC m=+1021.080584230" watchObservedRunningTime="2026-07-31 15:04:58.925487197 +0000 UTC m=+1021.083738709"
Jul 31 15:04:59.625257 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:04:59.625208    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:06.625482 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:06.625440    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:06.638258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:06.638205    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:06.946682 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:06.946652    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:44.733672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:44.733635    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:05:44.743755 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:05:44.743714    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:05:44.743755 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:05:44.743755 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:05:44.743755 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e" cmd=["/bin/sleep","15"]
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:44.743775    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" message="" err=<
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:05:44.743787    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main" containerID="cri-o://b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"
Jul 31 15:05:44.744098 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:44.743838    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main" containerID="cri-o://b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e" gracePeriod=60
Jul 31 15:05:44.936515 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:44.936472    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 31 15:05:45.093542 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.093512    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:45.116301 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.116265    2557 generic.go:356] "Generic (PLEG): container finished" podID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerID="b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e" exitCode=0
Jul 31 15:05:45.116471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.116323    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"
Jul 31 15:05:45.116471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.116364    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" event={"ID":"39e8aba9-ffa4-4990-be9b-411444c54c5e","Type":"ContainerDied","Data":"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"}
Jul 31 15:05:45.116471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.116427    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q" event={"ID":"39e8aba9-ffa4-4990-be9b-411444c54c5e","Type":"ContainerDied","Data":"6f644da84ff112574a901623e0fab669a075633f311f623e1b7420e576aa3b3e"}
Jul 31 15:05:45.116471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.116460    2557 scope.go:117] "RemoveContainer" containerID="b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"
Jul 31 15:05:45.125666 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.125647    2557 scope.go:117] "RemoveContainer" containerID="b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"
Jul 31 15:05:45.126040 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:05:45.126022    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e\": container with ID starting with b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e not found: ID does not exist" containerID="b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"
Jul 31 15:05:45.126093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.126051    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e"} err="failed to get container status \"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e\": rpc error: code = NotFound desc = could not find container \"b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e\": container with ID starting with b8584009979a024cd12811bcc20c69996ad6127b890a41ca5973faab73f3c17e not found: ID does not exist"
Jul 31 15:05:45.224525 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224487    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224571    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224601    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224637    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224675    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdqsb\" (UniqueName: \"kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224735    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir\") pod \"39e8aba9-ffa4-4990-be9b-411444c54c5e\" (UID: \"39e8aba9-ffa4-4990-be9b-411444c54c5e\") "
Jul 31 15:05:45.224922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224884    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:45.224922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.224900    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home" (OuterVolumeSpecName: "home") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:45.225070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.225044    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:45.225070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.225055    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.225157 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.225076    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.226871 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.226843    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm" (OuterVolumeSpecName: "dshm") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:45.226982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.226888    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:05:45.226982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.226921    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb" (OuterVolumeSpecName: "kube-api-access-wdqsb") pod "39e8aba9-ffa4-4990-be9b-411444c54c5e" (UID: "39e8aba9-ffa4-4990-be9b-411444c54c5e"). InnerVolumeSpecName "kube-api-access-wdqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:05:45.326316 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.326190    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.326316 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.326220    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39e8aba9-ffa4-4990-be9b-411444c54c5e-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.326316 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.326230    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdqsb\" (UniqueName: \"kubernetes.io/projected/39e8aba9-ffa4-4990-be9b-411444c54c5e-kube-api-access-wdqsb\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.326316 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.326268    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39e8aba9-ffa4-4990-be9b-411444c54c5e-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:05:45.447363 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.447320    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:05:45.456406 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:45.456372    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-6bg5q"]
Jul 31 15:05:46.248314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:05:46.248279    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" path="/var/lib/kubelet/pods/39e8aba9-ffa4-4990-be9b-411444c54c5e/volumes"
Jul 31 15:06:13.408091 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.408055    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:06:13.408741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.408577    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main"
Jul 31 15:06:13.408741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.408596    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main"
Jul 31 15:06:13.408741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.408711    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e8aba9-ffa4-4990-be9b-411444c54c5e" containerName="main"
Jul 31 15:06:13.411823 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.411799    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.415315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.414454    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 31 15:06:13.415927 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.415448    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-dnnq5\""
Jul 31 15:06:13.415927 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.415731    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 31 15:06:13.415927 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.415813    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 31 15:06:13.423017 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.422988    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:06:13.470983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.470934    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.471179 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.471062    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.471179 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.471098    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.471179 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.471166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.471327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.471201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.471327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.471230    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnhn\" (UniqueName: \"kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.571946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.571891    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.571946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.571952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572016    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572059    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572092    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnhn\" (UniqueName: \"kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572135    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572460 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572422    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572519 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572473    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.572519 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.572505    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.574452 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.574430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.574671 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.574651    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.580125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.580101    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnhn\" (UniqueName: \"kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.725834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.725794    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:13.870071 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:13.870017    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:06:13.872943 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:06:13.872915    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326a6ba2_abcc_4e5e_84f7_abb1ee8a49ec.slice/crio-2ed03534b410cbff7c2018e8329c41ef84b2d19e3f5e33c9961b7aa49a0fe8f2 WatchSource:0}: Error finding container 2ed03534b410cbff7c2018e8329c41ef84b2d19e3f5e33c9961b7aa49a0fe8f2: Status 404 returned error can't find the container with id 2ed03534b410cbff7c2018e8329c41ef84b2d19e3f5e33c9961b7aa49a0fe8f2
Jul 31 15:06:14.254190 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:14.254152    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" event={"ID":"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec","Type":"ContainerStarted","Data":"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"}
Jul 31 15:06:14.254401 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:14.254204    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" event={"ID":"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec","Type":"ContainerStarted","Data":"2ed03534b410cbff7c2018e8329c41ef84b2d19e3f5e33c9961b7aa49a0fe8f2"}
Jul 31 15:06:14.273576 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:14.273525    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" podStartSLOduration=1.27350603 podStartE2EDuration="1.27350603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:14.27246986 +0000 UTC m=+1096.430721418" watchObservedRunningTime="2026-07-31 15:06:14.27350603 +0000 UTC m=+1096.431757519"
Jul 31 15:06:14.726745 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:14.726703    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:23.726681 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:23.726635    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:23.739218 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:23.739196    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:24.313940 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:24.313910    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:57.933072 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:57.933038    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:06:57.943911 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:06:57.943869    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:06:57.943911 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:06:57.943911 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:06:57.943911 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3" cmd=["/bin/sleep","15"]
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:57.943934    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" message="" err=<
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:06:57.943951    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerName="main" containerID="cri-o://f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"
Jul 31 15:06:57.944332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:57.943995    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerName="main" containerID="cri-o://f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3" gracePeriod=60
Jul 31 15:06:58.302874 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.302847    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:58.373066 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373015    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgnhn\" (UniqueName: \"kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373091    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373157    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373266    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373340    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home\") pod \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\" (UID: \"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec\") "
Jul 31 15:06:58.373532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373417    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:58.373532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373405    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:58.373693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373619    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home" (OuterVolumeSpecName: "home") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:58.373693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373653    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.373693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.373672    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.375471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.375447    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm" (OuterVolumeSpecName: "dshm") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:58.375471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.375460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:58.375611 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.375537    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn" (OuterVolumeSpecName: "kube-api-access-qgnhn") pod "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" (UID: "326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec"). InnerVolumeSpecName "kube-api-access-qgnhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:58.468125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.468033    2557 generic.go:356] "Generic (PLEG): container finished" podID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerID="f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3" exitCode=0
Jul 31 15:06:58.468125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.468076    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"
Jul 31 15:06:58.468125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.468109    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" event={"ID":"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec","Type":"ContainerDied","Data":"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"}
Jul 31 15:06:58.468424 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.468160    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5" event={"ID":"326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec","Type":"ContainerDied","Data":"2ed03534b410cbff7c2018e8329c41ef84b2d19e3f5e33c9961b7aa49a0fe8f2"}
Jul 31 15:06:58.468424 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.468182    2557 scope.go:117] "RemoveContainer" containerID="f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"
Jul 31 15:06:58.474055 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.474032    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.474055 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.474054    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.474224 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.474064    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.474224 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.474073    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnhn\" (UniqueName: \"kubernetes.io/projected/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec-kube-api-access-qgnhn\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:06:58.478250 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.478212    2557 scope.go:117] "RemoveContainer" containerID="f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"
Jul 31 15:06:58.478626 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:06:58.478603    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3\": container with ID starting with f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3 not found: ID does not exist" containerID="f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"
Jul 31 15:06:58.478708 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.478633    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3"} err="failed to get container status \"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3\": rpc error: code = NotFound desc = could not find container \"f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3\": container with ID starting with f9a2a0ed903d55a17ed227459bd9c47a06f491caaac23bbe3cf90d929c1190b3 not found: ID does not exist"
Jul 31 15:06:58.489900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.489870    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:06:58.494383 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:06:58.494356    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fj84z5"]
Jul 31 15:07:00.248114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:00.248079    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" path="/var/lib/kubelet/pods/326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec/volumes"
Jul 31 15:07:21.112695 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.112615    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:21.113131 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.113014    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerName="main"
Jul 31 15:07:21.113131 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.113026    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerName="main"
Jul 31 15:07:21.113131 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.113088    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="326a6ba2-abcc-4e5e-84f7-abb1ee8a49ec" containerName="main"
Jul 31 15:07:21.117382 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.117359    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.121574 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.121544    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 31 15:07:21.121778 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.121587    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 31 15:07:21.121778 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.121553    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 31 15:07:21.121778 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.121663    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-rlmnl\""
Jul 31 15:07:21.128281 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.128253    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:21.271567 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271530    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklvq\" (UniqueName: \"kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.271772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.271772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271638    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.271772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271669    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.271932 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271797    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.271932 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.271840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372421    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklvq\" (UniqueName: \"kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372471    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372583    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372661    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372689    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.372989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.372972    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.373068 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.373045    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.373134 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.373096    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.374949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.374926    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.375107 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.375092    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.381007 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.380979    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklvq\" (UniqueName: \"kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.430198 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.430162    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:21.572135 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:21.572102    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:21.576335 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:07:21.576302    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14c7ac8_c74d_4c3b_93a6_eaff28e58e58.slice/crio-aed58e6e6b1464bdd58a834416b31abc742bd0c4514318724eb51e83b5a8c309 WatchSource:0}: Error finding container aed58e6e6b1464bdd58a834416b31abc742bd0c4514318724eb51e83b5a8c309: Status 404 returned error can't find the container with id aed58e6e6b1464bdd58a834416b31abc742bd0c4514318724eb51e83b5a8c309
Jul 31 15:07:22.583716 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:22.583676    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" event={"ID":"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58","Type":"ContainerStarted","Data":"aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79"}
Jul 31 15:07:22.584121 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:22.583732    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" event={"ID":"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58","Type":"ContainerStarted","Data":"aed58e6e6b1464bdd58a834416b31abc742bd0c4514318724eb51e83b5a8c309"}
Jul 31 15:07:22.635692 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:22.635631    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" podStartSLOduration=1.6356142 podStartE2EDuration="1.6356142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:07:22.634178421 +0000 UTC m=+1164.792429910" watchObservedRunningTime="2026-07-31 15:07:22.6356142 +0000 UTC m=+1164.793865747"
Jul 31 15:07:23.430954 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:23.430918    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:31.430810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:31.430773    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:31.443664 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:31.443634    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:31.636293 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:31.636263    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:54.416305 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.416269    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:54.425917 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:07:54.425879    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:07:54.425917 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:07:54.425917 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:07:54.425917 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79" cmd=["/bin/sleep","15"]
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.425932    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" message="" err=<
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:07:54.425946    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main" containerID="cri-o://aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79"
Jul 31 15:07:54.426207 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.425983    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main" containerID="cri-o://aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79" gracePeriod=60
Jul 31 15:07:54.626068 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.626032    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jul 31 15:07:54.730514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.730479    2557 generic.go:356] "Generic (PLEG): container finished" podID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerID="aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79" exitCode=0
Jul 31 15:07:54.730690 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.730554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" event={"ID":"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58","Type":"ContainerDied","Data":"aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79"}
Jul 31 15:07:54.770717 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.770683    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:54.869922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.869892    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.869941    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.869973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870025    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870074    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pklvq\" (UniqueName: \"kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870102    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir\") pod \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\" (UID: \"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58\") "
Jul 31 15:07:54.870358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870257    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home" (OuterVolumeSpecName: "home") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:54.870358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870293    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache" (OuterVolumeSpecName: "model-cache") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:54.870536 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.870475    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:54.872205 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.872182    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm" (OuterVolumeSpecName: "dshm") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:07:54.872325 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.872288    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq" (OuterVolumeSpecName: "kube-api-access-pklvq") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "kube-api-access-pklvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:07:54.872325 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.872312    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" (UID: "a14c7ac8-c74d-4c3b-93a6-eaff28e58e58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971330    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971359    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pklvq\" (UniqueName: \"kubernetes.io/projected/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-kube-api-access-pklvq\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971390    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971401    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971409    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:54.971412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:54.971416    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:07:55.736902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:55.736867    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2" event={"ID":"a14c7ac8-c74d-4c3b-93a6-eaff28e58e58","Type":"ContainerDied","Data":"aed58e6e6b1464bdd58a834416b31abc742bd0c4514318724eb51e83b5a8c309"}
Jul 31 15:07:55.736902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:55.736895    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"
Jul 31 15:07:55.737441 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:55.736930    2557 scope.go:117] "RemoveContainer" containerID="aea2cce97e0547b8412f79cb231ac7e0e232d18648566d827c566f9ae8a1cb79"
Jul 31 15:07:55.762488 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:55.762457    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:55.769336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:55.769306    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r4qc2"]
Jul 31 15:07:56.247222 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:56.247189    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" path="/var/lib/kubelet/pods/a14c7ac8-c74d-4c3b-93a6-eaff28e58e58/volumes"
Jul 31 15:07:58.229121 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:58.229089    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:07:58.230381 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:07:58.230356    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:08:23.383210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.383173    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:08:23.383637 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.383530    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main"
Jul 31 15:08:23.383637 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.383540    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main"
Jul 31 15:08:23.383637 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.383603    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14c7ac8-c74d-4c3b-93a6-eaff28e58e58" containerName="main"
Jul 31 15:08:23.386575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.386558    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.390356 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.390324    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 31 15:08:23.390520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.390325    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 31 15:08:23.390520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.390331    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-kc9xd\""
Jul 31 15:08:23.390520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.390413    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 31 15:08:23.399216 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.399192    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:08:23.431078 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.431078 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431084    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.431389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431115    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.431389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431210    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdqg\" (UniqueName: \"kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.431389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431279    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.431389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.431327    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532046    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532328 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532103    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdqg\" (UniqueName: \"kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532328 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532213    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532455 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532371    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532455 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532440    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532455 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532445    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532598 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532493    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532661 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532611    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.532712 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.532670    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.534638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.534610    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.534868 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.534851    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.540577 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.540551    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdqg\" (UniqueName: \"kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.697019 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.696986    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:23.838832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.838803    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:08:23.841207 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:08:23.841175    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0b010e_74d5_4edf_bca7_17d9da772fdf.slice/crio-1358e6b0b89eff960ede7d3467aaebfb83aca90c55cf807df2b9062c012d30b7 WatchSource:0}: Error finding container 1358e6b0b89eff960ede7d3467aaebfb83aca90c55cf807df2b9062c012d30b7: Status 404 returned error can't find the container with id 1358e6b0b89eff960ede7d3467aaebfb83aca90c55cf807df2b9062c012d30b7
Jul 31 15:08:23.865916 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:23.865889    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" event={"ID":"9b0b010e-74d5-4edf-bca7-17d9da772fdf","Type":"ContainerStarted","Data":"1358e6b0b89eff960ede7d3467aaebfb83aca90c55cf807df2b9062c012d30b7"}
Jul 31 15:08:24.872414 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:24.872376    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" event={"ID":"9b0b010e-74d5-4edf-bca7-17d9da772fdf","Type":"ContainerStarted","Data":"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"}
Jul 31 15:08:24.896148 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:24.896089    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" podStartSLOduration=1.8960723069999998 podStartE2EDuration="1.896072307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:08:24.892977286 +0000 UTC m=+1227.051228787" watchObservedRunningTime="2026-07-31 15:08:24.896072307 +0000 UTC m=+1227.054323798"
Jul 31 15:08:25.697930 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:25.697887    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:33.698144 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:33.698111    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:33.710662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:33.710636    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:33.923658 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:33.923628    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:58.844702 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:58.844612    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:08:58.854253 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:08:58.854191    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:08:58.854253 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:08:58.854253 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:08:58.854253 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32" cmd=["/bin/sleep","15"]
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:58.854268    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" message="" err=<
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:08:58.854286    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main" containerID="cri-o://f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"
Jul 31 15:08:58.854602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:58.854345    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main" containerID="cri-o://f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32" gracePeriod=60
Jul 31 15:08:58.913663 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:58.913620    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.35:8000/health\": dial tcp 10.132.0.35:8000: connect: connection refused"
Jul 31 15:08:59.201450 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.201428    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:08:59.354306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354275    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354322    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsdqg\" (UniqueName: \"kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354380    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354408    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354447    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354745 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354530    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir\") pod \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\" (UID: \"9b0b010e-74d5-4edf-bca7-17d9da772fdf\") "
Jul 31 15:08:59.354745 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354598    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home" (OuterVolumeSpecName: "home") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:59.354745 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354739    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:59.354913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354767    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:59.354913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354844    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:08:59.354913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354859    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:08:59.354913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.354871    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:08:59.356652 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.356628    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm" (OuterVolumeSpecName: "dshm") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:59.356759 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.356672    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:08:59.356759 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.356711    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg" (OuterVolumeSpecName: "kube-api-access-hsdqg") pod "9b0b010e-74d5-4edf-bca7-17d9da772fdf" (UID: "9b0b010e-74d5-4edf-bca7-17d9da772fdf"). InnerVolumeSpecName "kube-api-access-hsdqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:08:59.455858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.455822    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdqg\" (UniqueName: \"kubernetes.io/projected/9b0b010e-74d5-4edf-bca7-17d9da772fdf-kube-api-access-hsdqg\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:08:59.455858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.455853    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b0b010e-74d5-4edf-bca7-17d9da772fdf-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:08:59.455858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:08:59.455864    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b0b010e-74d5-4edf-bca7-17d9da772fdf-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:09:00.031417 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.031382    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerID="f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32" exitCode=0
Jul 31 15:09:00.031824 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.031426    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"
Jul 31 15:09:00.031824 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.031473    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" event={"ID":"9b0b010e-74d5-4edf-bca7-17d9da772fdf","Type":"ContainerDied","Data":"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"}
Jul 31 15:09:00.031824 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.031519    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z" event={"ID":"9b0b010e-74d5-4edf-bca7-17d9da772fdf","Type":"ContainerDied","Data":"1358e6b0b89eff960ede7d3467aaebfb83aca90c55cf807df2b9062c012d30b7"}
Jul 31 15:09:00.031824 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.031542    2557 scope.go:117] "RemoveContainer" containerID="f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"
Jul 31 15:09:00.040300 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.040280    2557 scope.go:117] "RemoveContainer" containerID="f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"
Jul 31 15:09:00.040547 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:09:00.040528    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32\": container with ID starting with f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32 not found: ID does not exist" containerID="f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"
Jul 31 15:09:00.040604 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.040556    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32"} err="failed to get container status \"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32\": rpc error: code = NotFound desc = could not find container \"f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32\": container with ID starting with f0b5303cc0df784cdd4fbfc580b98a5217272a625bd9b0bab9e20e6df1bbaf32 not found: ID does not exist"
Jul 31 15:09:00.056328 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.056297    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:09:00.064391 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.064368    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-2hn9z"]
Jul 31 15:09:00.247769 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:00.247732    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" path="/var/lib/kubelet/pods/9b0b010e-74d5-4edf-bca7-17d9da772fdf/volumes"
Jul 31 15:09:40.098884 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.098845    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:09:40.099749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.099725    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main"
Jul 31 15:09:40.099749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.099751    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main"
Jul 31 15:09:40.099894 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.099877    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0b010e-74d5-4edf-bca7-17d9da772fdf" containerName="main"
Jul 31 15:09:40.103341 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.103323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.106790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.106757    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-7f8hj\""
Jul 31 15:09:40.106922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.106801    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 31 15:09:40.106922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.106758    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 31 15:09:40.106922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.106878    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 31 15:09:40.110044 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.110018    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:09:40.206210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206172    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.206410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206220    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kj9q\" (UniqueName: \"kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.206410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206291    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.206410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.206410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206351    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.206410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.206401    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.307639 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307604    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.307639 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307644    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kj9q\" (UniqueName: \"kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.307898 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307690    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.307898 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.307898 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307869    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.308071 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307945    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.308071 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.307977    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.308071 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.308051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.308317 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.308294    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.310107 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.310085    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.310379 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.310362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.316054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.316028    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kj9q\" (UniqueName: \"kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-mtb85\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.415621 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.415529    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:40.551469 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:40.551439    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:09:40.554065 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:09:40.554037    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5aacaab_57b0_4647_a7e6_b4d2d86312f7.slice/crio-8609840ac22f9935cc973315afebc1d69c19b558e68fbd525530aea0d66d29b7 WatchSource:0}: Error finding container 8609840ac22f9935cc973315afebc1d69c19b558e68fbd525530aea0d66d29b7: Status 404 returned error can't find the container with id 8609840ac22f9935cc973315afebc1d69c19b558e68fbd525530aea0d66d29b7
Jul 31 15:09:41.225307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:41.225268    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" event={"ID":"a5aacaab-57b0-4647-a7e6-b4d2d86312f7","Type":"ContainerStarted","Data":"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"}
Jul 31 15:09:41.225722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:41.225328    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" event={"ID":"a5aacaab-57b0-4647-a7e6-b4d2d86312f7","Type":"ContainerStarted","Data":"8609840ac22f9935cc973315afebc1d69c19b558e68fbd525530aea0d66d29b7"}
Jul 31 15:09:41.255795 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:41.255741    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" podStartSLOduration=1.255724049 podStartE2EDuration="1.255724049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:09:41.253924328 +0000 UTC m=+1303.412175818" watchObservedRunningTime="2026-07-31 15:09:41.255724049 +0000 UTC m=+1303.413975542"
Jul 31 15:09:41.416306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:41.416229    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:50.416503 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:50.416466    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:50.429010 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:50.428987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:09:51.281838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:09:51.281810    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:10:14.222563 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.222530    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:10:14.233091 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:10:14.233030    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:10:14.233091 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:10:14.233091 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:10:14.233091 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c" cmd=["/bin/sleep","15"]
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.233113    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" message="" err=<
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:10:14.233176    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main" containerID="cri-o://4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"
Jul 31 15:10:14.233443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.233360    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main" containerID="cri-o://4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c" gracePeriod=60
Jul 31 15:10:14.271454 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.271415    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 31 15:10:14.583388 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.583361    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:10:14.616780 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616747    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.616780 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616788    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.617034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616807    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.617034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616838    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kj9q\" (UniqueName: \"kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.617034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616855    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.617034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.616943    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home\") pod \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\" (UID: \"a5aacaab-57b0-4647-a7e6-b4d2d86312f7\") "
Jul 31 15:10:14.617412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.617084    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:14.617412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.617088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:14.617412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.617333    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:14.617412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.617353    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:14.617627 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.617336    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home" (OuterVolumeSpecName: "home") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:14.619329 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.619298    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm" (OuterVolumeSpecName: "dshm") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:14.619429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.619380    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q" (OuterVolumeSpecName: "kube-api-access-7kj9q") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "kube-api-access-7kj9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:14.619664 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.619643    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5aacaab-57b0-4647-a7e6-b4d2d86312f7" (UID: "a5aacaab-57b0-4647-a7e6-b4d2d86312f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:14.718496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.718452    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:14.718496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.718484    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:14.718496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.718493    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kj9q\" (UniqueName: \"kubernetes.io/projected/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-kube-api-access-7kj9q\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:14.718496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:14.718505    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aacaab-57b0-4647-a7e6-b4d2d86312f7-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:10:15.386228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.386190    2557 generic.go:356] "Generic (PLEG): container finished" podID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerID="4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c" exitCode=0
Jul 31 15:10:15.386681 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.386260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" event={"ID":"a5aacaab-57b0-4647-a7e6-b4d2d86312f7","Type":"ContainerDied","Data":"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"}
Jul 31 15:10:15.386681 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.386308    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85" event={"ID":"a5aacaab-57b0-4647-a7e6-b4d2d86312f7","Type":"ContainerDied","Data":"8609840ac22f9935cc973315afebc1d69c19b558e68fbd525530aea0d66d29b7"}
Jul 31 15:10:15.386681 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.386331    2557 scope.go:117] "RemoveContainer" containerID="4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"
Jul 31 15:10:15.386681 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.386265    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"
Jul 31 15:10:15.395725 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.395707    2557 scope.go:117] "RemoveContainer" containerID="4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"
Jul 31 15:10:15.396027 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:10:15.396009    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c\": container with ID starting with 4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c not found: ID does not exist" containerID="4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"
Jul 31 15:10:15.396074 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.396036    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c"} err="failed to get container status \"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c\": rpc error: code = NotFound desc = could not find container \"4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c\": container with ID starting with 4ddd3d6cc2cbf88f3d67c09eaf2cc24bc9645f6437e75a379286254b2d42933c not found: ID does not exist"
Jul 31 15:10:15.410918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.410879    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:10:15.414995 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:15.414966    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-mtb85"]
Jul 31 15:10:16.247630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:16.247599    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" path="/var/lib/kubelet/pods/a5aacaab-57b0-4647-a7e6-b4d2d86312f7/volumes"
Jul 31 15:10:38.355388 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.355348    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:10:38.355858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.355833    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main"
Jul 31 15:10:38.355858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.355848    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main"
Jul 31 15:10:38.355972 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.355924    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5aacaab-57b0-4647-a7e6-b4d2d86312f7" containerName="main"
Jul 31 15:10:38.359119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.359099    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.362609 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.362583    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 31 15:10:38.362742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.362682    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-pdjxs\""
Jul 31 15:10:38.362742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.362706    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 31 15:10:38.362742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.362706    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-pt24g\""
Jul 31 15:10:38.362742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.362681    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 31 15:10:38.373491 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.373462    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:10:38.425304 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425268    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkjn\" (UniqueName: \"kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.425486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425347    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.425486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425378    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.425486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425394    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.425486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425426    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.425486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.425467    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526420 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526373    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkjn\" (UniqueName: \"kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526436    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526462    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526499    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.526966 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526945    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.527020 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.526986    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.527057 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.527015    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.528799 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.528779    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.529051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.529031    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.534753 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.534730    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkjn\" (UniqueName: \"kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.668814 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.668718    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:38.818699 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.818667    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:10:38.821377 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:10:38.821345    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf067d1_35dc_455d_bb3e_2831ea6e9b6b.slice/crio-cdf2335267104d1458a646709d1780955e46f601ebaa095431446cea46740ca0 WatchSource:0}: Error finding container cdf2335267104d1458a646709d1780955e46f601ebaa095431446cea46740ca0: Status 404 returned error can't find the container with id cdf2335267104d1458a646709d1780955e46f601ebaa095431446cea46740ca0
Jul 31 15:10:38.823578 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:38.823563    2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:10:39.505515 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:39.505470    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerStarted","Data":"cdf2335267104d1458a646709d1780955e46f601ebaa095431446cea46740ca0"}
Jul 31 15:10:41.517438 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:41.517401    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerStarted","Data":"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"}
Jul 31 15:10:41.517951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:41.517523    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:42.524008 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:42.523965    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerStarted","Data":"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"}
Jul 31 15:10:42.543612 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:42.543555    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" podStartSLOduration=2.330524318 podStartE2EDuration="4.543537053s" totalImagesPullingTime="2.213012735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:42.54237676 +0000 UTC m=+1364.700628249" watchObservedRunningTime="2026-07-31 15:10:42.543537053 +0000 UTC m=+1364.701788542"
Jul 31 15:10:42.670061 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:42.670027    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:48.669356 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:48.669327    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:48.681819 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:48.681793    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:10:49.566971 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:10:49.566940    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:11:00.573257 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:00.573210    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:11:12.484624 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.484592    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:11:12.493853 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:11:12.493815    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:11:12.493853 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.493853 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.493853 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63" cmd=["/bin/sleep","15"]
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.493870    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" message="" err=<
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:11:12.493882    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main" containerID="cri-o://4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"
Jul 31 15:11:12.494114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.493917    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main" containerID="cri-o://4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63" gracePeriod=60
Jul 31 15:11:12.557065 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.557019    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jul 31 15:11:12.710316 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.710282    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="llm-d-routing-sidecar" containerID="cri-o://281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f" gracePeriod=60
Jul 31 15:11:12.960600 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:12.960569    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:11:13.035994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.035958    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036009    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgkjn\" (UniqueName: \"kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036036    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036077    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036110    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036156    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir\") pod \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\" (UID: \"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b\") "
Jul 31 15:11:13.036473 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036359    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.036473 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home" (OuterVolumeSpecName: "home") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.036587 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.036480    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.038278 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.038218    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm" (OuterVolumeSpecName: "dshm") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:13.038278 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.038232    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn" (OuterVolumeSpecName: "kube-api-access-jgkjn") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "kube-api-access-jgkjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:13.038432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.038325    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" (UID: "cbf067d1-35dc-455d-bb3e-2831ea6e9b6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137350    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137380    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgkjn\" (UniqueName: \"kubernetes.io/projected/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-kube-api-access-jgkjn\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137391    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137401    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137409    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.137432 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.137416    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:11:13.672229 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672198    2557 generic.go:356] "Generic (PLEG): container finished" podID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63" exitCode=0
Jul 31 15:11:13.672229 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672221    2557 generic.go:356] "Generic (PLEG): container finished" podID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerID="281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f" exitCode=0
Jul 31 15:11:13.672732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672265    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"
Jul 31 15:11:13.672732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672271    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerDied","Data":"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"}
Jul 31 15:11:13.672732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672331    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerDied","Data":"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"}
Jul 31 15:11:13.672732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672348    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr" event={"ID":"cbf067d1-35dc-455d-bb3e-2831ea6e9b6b","Type":"ContainerDied","Data":"cdf2335267104d1458a646709d1780955e46f601ebaa095431446cea46740ca0"}
Jul 31 15:11:13.672732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.672368    2557 scope.go:117] "RemoveContainer" containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"
Jul 31 15:11:13.681787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.681765    2557 scope.go:117] "RemoveContainer" containerID="281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"
Jul 31 15:11:13.689217 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689197    2557 scope.go:117] "RemoveContainer" containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"
Jul 31 15:11:13.689496 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:11:13.689475    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63\": container with ID starting with 4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63 not found: ID does not exist" containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"
Jul 31 15:11:13.689550 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689506    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"} err="failed to get container status \"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63\": rpc error: code = NotFound desc = could not find container \"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63\": container with ID starting with 4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63 not found: ID does not exist"
Jul 31 15:11:13.689550 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689524    2557 scope.go:117] "RemoveContainer" containerID="281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"
Jul 31 15:11:13.689744 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:11:13.689730    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f\": container with ID starting with 281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f not found: ID does not exist" containerID="281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"
Jul 31 15:11:13.689785 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689748    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"} err="failed to get container status \"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f\": rpc error: code = NotFound desc = could not find container \"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f\": container with ID starting with 281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f not found: ID does not exist"
Jul 31 15:11:13.689785 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689760    2557 scope.go:117] "RemoveContainer" containerID="4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"
Jul 31 15:11:13.689982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689964    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63"} err="failed to get container status \"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63\": rpc error: code = NotFound desc = could not find container \"4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63\": container with ID starting with 4dd6f2e104fe115ede1cb0cebfa38d9c0e2d633a73152c9690cbc76778445c63 not found: ID does not exist"
Jul 31 15:11:13.689982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.689980    2557 scope.go:117] "RemoveContainer" containerID="281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"
Jul 31 15:11:13.690173 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.690157    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f"} err="failed to get container status \"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f\": rpc error: code = NotFound desc = could not find container \"281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f\": container with ID starting with 281a6806491702042277260f7f6eda3cd8015ce13a470decd72087766edc4e7f not found: ID does not exist"
Jul 31 15:11:13.694849 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.694794    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:11:13.696361 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:13.696339    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-ppqjr"]
Jul 31 15:11:14.248442 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:14.248409    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" path="/var/lib/kubelet/pods/cbf067d1-35dc-455d-bb3e-2831ea6e9b6b/volumes"
Jul 31 15:11:48.452153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452116    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452517    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main"
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452529    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main"
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452540    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="llm-d-routing-sidecar"
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452546    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="llm-d-routing-sidecar"
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452603    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="main"
Jul 31 15:11:48.452665 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.452610    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf067d1-35dc-455d-bb3e-2831ea6e9b6b" containerName="llm-d-routing-sidecar"
Jul 31 15:11:48.455913 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.455892    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.458341 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.458314    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 31 15:11:48.458497 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.458470    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-mwbbs\""
Jul 31 15:11:48.459346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.459322    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 31 15:11:48.459452 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.459350    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-n9hbs\""
Jul 31 15:11:48.459515 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.459453    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 31 15:11:48.468387 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.468361    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:11:48.546084 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.546320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546104    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.546320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546184    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.546320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546217    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qpfc\" (UniqueName: \"kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.546320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546312    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.546497 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.546337    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.646876 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.646840    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.646876 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.646884    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647120 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647004    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647120 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647040    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qpfc\" (UniqueName: \"kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647120 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647118    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647150    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647286    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647437 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647392    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.647517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.647498    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.649221 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.649204    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.649413 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.649399    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.658538 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.658516    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qpfc\" (UniqueName: \"kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.767137 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.767046    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:48.903600 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:48.903574    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:11:48.906251 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:11:48.906210    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797ed0b_5d89_4e44_ade0_10d9bcf0d62d.slice/crio-33fe5cdc4acaddf6df6f545f95cc40bd51179a97f315955e887ebf8af6ee5a73 WatchSource:0}: Error finding container 33fe5cdc4acaddf6df6f545f95cc40bd51179a97f315955e887ebf8af6ee5a73: Status 404 returned error can't find the container with id 33fe5cdc4acaddf6df6f545f95cc40bd51179a97f315955e887ebf8af6ee5a73
Jul 31 15:11:49.846591 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:49.846554    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerStarted","Data":"e03ad3deaca6674aa725aaa64723d0293a640a2d728110231b8561c3d17a010f"}
Jul 31 15:11:49.846990 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:49.846620    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerStarted","Data":"33fe5cdc4acaddf6df6f545f95cc40bd51179a97f315955e887ebf8af6ee5a73"}
Jul 31 15:11:49.846990 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:49.846650    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:50.859181 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:50.859144    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerStarted","Data":"9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f"}
Jul 31 15:11:50.880858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:50.880800    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" podStartSLOduration=2.880780551 podStartE2EDuration="2.880780551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:50.87927692 +0000 UTC m=+1433.037528410" watchObservedRunningTime="2026-07-31 15:11:50.880780551 +0000 UTC m=+1433.039032044"
Jul 31 15:11:51.767985 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:51.767951    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:58.767512 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:58.767469    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:58.780818 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:58.780788    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:58.780986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:58.780936    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:11:58.907637 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:11:58.907604    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:12:56.747258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:56.747200    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:12:56.757899 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:12:56.757857    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:12:56.757899 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.757899 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.757899 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f" cmd=["/bin/sleep","15"]
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:56.757915    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" message="" err=<
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:12:56.757928    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main" containerID="cri-o://9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f"
Jul 31 15:12:56.758280 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:56.757977    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main" containerID="cri-o://9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f" gracePeriod=60
Jul 31 15:12:56.896963 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:56.896916    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8001/health\": dial tcp 10.132.0.38:8001: connect: connection refused"
Jul 31 15:12:56.974612 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:56.974580    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="llm-d-routing-sidecar" containerID="cri-o://e03ad3deaca6674aa725aaa64723d0293a640a2d728110231b8561c3d17a010f" gracePeriod=60
Jul 31 15:12:57.175165 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.175128    2557 generic.go:356] "Generic (PLEG): container finished" podID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerID="9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f" exitCode=0
Jul 31 15:12:57.175165 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.175159    2557 generic.go:356] "Generic (PLEG): container finished" podID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerID="e03ad3deaca6674aa725aaa64723d0293a640a2d728110231b8561c3d17a010f" exitCode=0
Jul 31 15:12:57.175388 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.175193    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerDied","Data":"9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f"}
Jul 31 15:12:57.175388 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.175264    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerDied","Data":"e03ad3deaca6674aa725aaa64723d0293a640a2d728110231b8561c3d17a010f"}
Jul 31 15:12:57.230674 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.230648    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:12:57.263748 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263713    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.263748 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263754    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.263992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.263992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263864    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.263992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263906    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qpfc\" (UniqueName: \"kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.263992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.263961    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir\") pod \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\" (UID: \"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d\") "
Jul 31 15:12:57.264212 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.264168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home" (OuterVolumeSpecName: "home") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.264307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.264282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.264365 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.264330    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.264428 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.264412    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.266054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.266027    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:12:57.266507 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.266480    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm" (OuterVolumeSpecName: "dshm") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:12:57.266507 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.266483    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc" (OuterVolumeSpecName: "kube-api-access-5qpfc") pod "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" (UID: "8797ed0b-5d89-4e44-ade0-10d9bcf0d62d"). InnerVolumeSpecName "kube-api-access-5qpfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:12:57.364826 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.364732    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.364826 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.364766    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qpfc\" (UniqueName: \"kubernetes.io/projected/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-kube-api-access-5qpfc\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.364826 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.364781    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.364826 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.364791    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:57.364826 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:57.364803    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:12:58.182077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.182041    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r" event={"ID":"8797ed0b-5d89-4e44-ade0-10d9bcf0d62d","Type":"ContainerDied","Data":"33fe5cdc4acaddf6df6f545f95cc40bd51179a97f315955e887ebf8af6ee5a73"}
Jul 31 15:12:58.182077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.182083    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"
Jul 31 15:12:58.182630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.182117    2557 scope.go:117] "RemoveContainer" containerID="9cc7b58556693c9b80479c6ba7f379a7f068b78955e56761e9fb771c6b749b2f"
Jul 31 15:12:58.194054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.194027    2557 scope.go:117] "RemoveContainer" containerID="e03ad3deaca6674aa725aaa64723d0293a640a2d728110231b8561c3d17a010f"
Jul 31 15:12:58.206268 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.206225    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:12:58.212986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.212962    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-86m6r"]
Jul 31 15:12:58.249155 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.249128    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" path="/var/lib/kubelet/pods/8797ed0b-5d89-4e44-ade0-10d9bcf0d62d/volumes"
Jul 31 15:12:58.260792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.260740    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:12:58.265800 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:12:58.265770    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:13:24.011073 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.010991    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011363    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main"
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011378    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main"
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011412    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="llm-d-routing-sidecar"
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011419    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="llm-d-routing-sidecar"
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011494    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="llm-d-routing-sidecar"
Jul 31 15:13:24.011736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.011510    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="8797ed0b-5d89-4e44-ade0-10d9bcf0d62d" containerName="main"
Jul 31 15:13:24.016088 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.016070    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.018618 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.018594    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 31 15:13:24.018772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.018754    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:13:24.019602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.019589    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 31 15:13:24.019768 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.019756    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 31 15:13:24.019849 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.019816    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-h8q2s\""
Jul 31 15:13:24.019921 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.019878    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 31 15:13:24.025344 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.025316    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:13:24.092067 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092024    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092067 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092066    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4qk\" (UniqueName: \"kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092121    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092138    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092212    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.092353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.092257    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193527 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193488    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193527 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193529    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193782 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193557    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193782 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193686    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193782 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193944 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193863    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193944 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193901    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.193944 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193927    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4qk\" (UniqueName: \"kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.194095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.193963    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.194167 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.194139    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.194257 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.194217    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.194324 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.194297    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.194441 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.194424    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.196025 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.196004    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.196145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.196081    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.202276 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.202230    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4qk\" (UniqueName: \"kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk\") pod \"tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.327037 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.326946    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:24.468282 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:24.468254    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:13:24.469952 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:13:24.469930    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4700d4c3_4ea4_4f27_97cf_fc9184cea7aa.slice/crio-537ce62efc5f1cef5897e45c4fc720f55f16fab6d71a3325d2bdcc258b19a3b8 WatchSource:0}: Error finding container 537ce62efc5f1cef5897e45c4fc720f55f16fab6d71a3325d2bdcc258b19a3b8: Status 404 returned error can't find the container with id 537ce62efc5f1cef5897e45c4fc720f55f16fab6d71a3325d2bdcc258b19a3b8
Jul 31 15:13:25.306061 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:25.306016    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerStarted","Data":"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"}
Jul 31 15:13:25.306061 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:25.306069    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerStarted","Data":"537ce62efc5f1cef5897e45c4fc720f55f16fab6d71a3325d2bdcc258b19a3b8"}
Jul 31 15:13:27.317895 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:27.317859    2557 generic.go:356] "Generic (PLEG): container finished" podID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerID="c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b" exitCode=0
Jul 31 15:13:27.318299 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:27.317932    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerDied","Data":"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"}
Jul 31 15:13:28.324487 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:28.324452    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerStarted","Data":"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"}
Jul 31 15:13:28.327930 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:28.327910    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:28.357728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:28.357672    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" podStartSLOduration=3.054342922 podStartE2EDuration="5.357656293s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.303313371s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:13:28.357199337 +0000 UTC m=+1530.515450838" watchObservedRunningTime="2026-07-31 15:13:28.357656293 +0000 UTC m=+1530.515907783"
Jul 31 15:13:34.327738 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:34.327705    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:34.340348 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:34.340321    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:13:34.362195 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:13:34.362153    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:14:26.099205 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.099172    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:14:26.102801 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.102779    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.105518 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.105494    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Jul 31 15:14:26.105634 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.105548    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Jul 31 15:14:26.106436 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.106418    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Jul 31 15:14:26.106593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.106442    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-2nmgb\""
Jul 31 15:14:26.111814 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.111794    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:14:26.242070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242032    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.242070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.242395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242103    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4l5x\" (UniqueName: \"kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.242395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242167    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.242395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242181    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.242395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.242227    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342757 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342724    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342768    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342791    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4l5x\" (UniqueName: \"kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342847    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.342958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.342864    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.343290 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.343263    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.343403 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.343298    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.343403 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.343362    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.345106 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.345077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.345262 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.345224    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.351870 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.351819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4l5x\" (UniqueName: \"kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x\") pod \"test-div-v1-kserve-fb566d667-cp77r\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.415178 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.415138    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:26.556352 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.556326    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:14:26.559035 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:14:26.559004    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3285b8_2799_4fec_80e0_fa0723e8042d.slice/crio-221fe9a3f35cd3d4c08905299e1103f25c44fddcd5a9f5ce62a84cc39b279d73 WatchSource:0}: Error finding container 221fe9a3f35cd3d4c08905299e1103f25c44fddcd5a9f5ce62a84cc39b279d73: Status 404 returned error can't find the container with id 221fe9a3f35cd3d4c08905299e1103f25c44fddcd5a9f5ce62a84cc39b279d73
Jul 31 15:14:26.601197 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:26.601157    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" event={"ID":"9b3285b8-2799-4fec-80e0-fa0723e8042d","Type":"ContainerStarted","Data":"221fe9a3f35cd3d4c08905299e1103f25c44fddcd5a9f5ce62a84cc39b279d73"}
Jul 31 15:14:28.612745 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:28.612701    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" event={"ID":"9b3285b8-2799-4fec-80e0-fa0723e8042d","Type":"ContainerStarted","Data":"b1382c8d6157041c043ccae38592e032972e93c64a3e010d54e1c2c420745490"}
Jul 31 15:14:28.637042 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:28.636810    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" podStartSLOduration=1.5926183919999999 podStartE2EDuration="2.636787789s" totalImagesPullingTime="1.044169397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:28.632801619 +0000 UTC m=+1590.791053132" watchObservedRunningTime="2026-07-31 15:14:28.636787789 +0000 UTC m=+1590.795039279"
Jul 31 15:14:29.415629 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:29.415594    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:34.298807 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.298773    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:14:34.302372 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.302344    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.304766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.304747    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Jul 31 15:14:34.311456 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.311432    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:14:34.418783 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.418736    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.418959 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.418814    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.418959 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.418840    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.418959 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.418931    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.419149 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.418972    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.419149 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.419013    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd55v\" (UniqueName: \"kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.519958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.519919    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd55v\" (UniqueName: \"kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.519958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.519960    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520186 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.519981    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520186 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520089    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520200    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520278    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520426 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520318    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520579 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520556    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.520663 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.520562    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.522684 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.522658    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.522846 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.522741    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.527567 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.527543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd55v\" (UniqueName: \"kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v\") pod \"test-div-v2-kserve-5f6fdb6b77-dnddc\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.613506 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.613395    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:34.755973 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:34.755946    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:14:34.758171 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:14:34.758139    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd220041a_133c_4b7b_aaae_847b918fc809.slice/crio-4205580b8d3244e98452dbda5a5c28026812058dc8eeaf0d8255442943b3ab29 WatchSource:0}: Error finding container 4205580b8d3244e98452dbda5a5c28026812058dc8eeaf0d8255442943b3ab29: Status 404 returned error can't find the container with id 4205580b8d3244e98452dbda5a5c28026812058dc8eeaf0d8255442943b3ab29
Jul 31 15:14:35.648231 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:35.648194    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" event={"ID":"d220041a-133c-4b7b-aaae-847b918fc809","Type":"ContainerStarted","Data":"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"}
Jul 31 15:14:35.648650 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:35.648260    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" event={"ID":"d220041a-133c-4b7b-aaae-847b918fc809","Type":"ContainerStarted","Data":"4205580b8d3244e98452dbda5a5c28026812058dc8eeaf0d8255442943b3ab29"}
Jul 31 15:14:35.668318 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:35.668260    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" podStartSLOduration=1.668232099 podStartE2EDuration="1.668232099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:35.666156381 +0000 UTC m=+1597.824407873" watchObservedRunningTime="2026-07-31 15:14:35.668232099 +0000 UTC m=+1597.826483590"
Jul 31 15:14:36.415735 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:36.415682    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:36.428389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:36.428359    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:36.614275 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:36.614221    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:36.664062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:36.664036    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:14:44.614303 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:44.614263    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:44.630044 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:44.630011    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:44.703843 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:44.703808    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:14:56.753148 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:56.753061    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:14:56.762695 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:14:56.762659    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:14:56.762695 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.762695 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.762695 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71" cmd=["/bin/sleep","15"]
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:56.762713    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" message="" err=<
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:14:56.762725    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="main" containerID="cri-o://0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"
Jul 31 15:14:56.763027 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:56.762772    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="main" containerID="cri-o://0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71" gracePeriod=60
Jul 31 15:14:57.130483 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.130449    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:14:57.227614 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227583    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.227810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227631    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.227810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227673    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q4qk\" (UniqueName: \"kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.227810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227696    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.227810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227725    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.227810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227759    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.228079 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227811    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.228079 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227862    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir\") pod \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\" (UID: \"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa\") "
Jul 31 15:14:57.228079 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.227949    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.228079 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.228059    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home" (OuterVolumeSpecName: "home") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.228327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.228170    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:14:57.228327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.228196    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.228327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.228217    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.228327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.228179    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.230107 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.230077    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:14:57.230219 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.230102    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk" (OuterVolumeSpecName: "kube-api-access-2q4qk") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "kube-api-access-2q4qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:14:57.230433 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.230411    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm" (OuterVolumeSpecName: "dshm") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.249003 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.248966    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" (UID: "4700d4c3-4ea4-4f27-97cf-fc9184cea7aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329436    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329465    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4qk\" (UniqueName: \"kubernetes.io/projected/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kube-api-access-2q4qk\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329475    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329484    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329492    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.329522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.329499    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:14:57.757330 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.757292    2557 generic.go:356] "Generic (PLEG): container finished" podID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerID="0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71" exitCode=0
Jul 31 15:14:57.757737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.757335    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerDied","Data":"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"}
Jul 31 15:14:57.757737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.757343    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"
Jul 31 15:14:57.757737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.757379    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf" event={"ID":"4700d4c3-4ea4-4f27-97cf-fc9184cea7aa","Type":"ContainerDied","Data":"537ce62efc5f1cef5897e45c4fc720f55f16fab6d71a3325d2bdcc258b19a3b8"}
Jul 31 15:14:57.757737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.757401    2557 scope.go:117] "RemoveContainer" containerID="0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"
Jul 31 15:14:57.774145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.773988    2557 scope.go:117] "RemoveContainer" containerID="c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"
Jul 31 15:14:57.782203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.782182    2557 scope.go:117] "RemoveContainer" containerID="0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"
Jul 31 15:14:57.782551 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:14:57.782530    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71\": container with ID starting with 0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71 not found: ID does not exist" containerID="0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"
Jul 31 15:14:57.782630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.782561    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71"} err="failed to get container status \"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71\": rpc error: code = NotFound desc = could not find container \"0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71\": container with ID starting with 0c715421df0c380ce965bc0fa4d4b84bb9d09743ca4514aec181b9c628d92a71 not found: ID does not exist"
Jul 31 15:14:57.782630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.782582    2557 scope.go:117] "RemoveContainer" containerID="c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"
Jul 31 15:14:57.782858 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:14:57.782840    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b\": container with ID starting with c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b not found: ID does not exist" containerID="c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"
Jul 31 15:14:57.782912 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.782865    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b"} err="failed to get container status \"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b\": rpc error: code = NotFound desc = could not find container \"c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b\": container with ID starting with c46bf681e848396a0995ed3e2c5451bf7fead7ae456d089f8997a45dfd3f6f1b not found: ID does not exist"
Jul 31 15:14:57.785592 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.785568    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:14:57.792285 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:57.792232    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6cdfc86fbc-t8lrf"]
Jul 31 15:14:58.248441 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:14:58.248414    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" path="/var/lib/kubelet/pods/4700d4c3-4ea4-4f27-97cf-fc9184cea7aa/volumes"
Jul 31 15:15:00.689158 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:00.689126    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:15:00.694439 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:00.694411    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:15:15.709984 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:15.709937    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" podUID="9b3285b8-2799-4fec-80e0-fa0723e8042d" containerName="main" containerID="cri-o://b1382c8d6157041c043ccae38592e032972e93c64a3e010d54e1c2c420745490" gracePeriod=45
Jul 31 15:15:15.769163 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:15.769127    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" podUID="d220041a-133c-4b7b-aaae-847b918fc809" containerName="main" containerID="cri-o://ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2" gracePeriod=45
Jul 31 15:15:15.842851 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:15.842820    2557 generic.go:356] "Generic (PLEG): container finished" podID="9b3285b8-2799-4fec-80e0-fa0723e8042d" containerID="b1382c8d6157041c043ccae38592e032972e93c64a3e010d54e1c2c420745490" exitCode=0
Jul 31 15:15:15.843018 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:15.842867    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" event={"ID":"9b3285b8-2799-4fec-80e0-fa0723e8042d","Type":"ContainerDied","Data":"b1382c8d6157041c043ccae38592e032972e93c64a3e010d54e1c2c420745490"}
Jul 31 15:15:16.027177 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.027149    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:15:16.055120 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.055093    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:15:16.208496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208465    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208509    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208536    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd55v\" (UniqueName: \"kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208568    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208645    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208670    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208702    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208789    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208818    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs\") pod \"d220041a-133c-4b7b-aaae-847b918fc809\" (UID: \"d220041a-133c-4b7b-aaae-847b918fc809\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208823    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208827    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache" (OuterVolumeSpecName: "model-cache") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208856    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4l5x\" (UniqueName: \"kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208880    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache\") pod \"9b3285b8-2799-4fec-80e0-fa0723e8042d\" (UID: \"9b3285b8-2799-4fec-80e0-fa0723e8042d\") "
Jul 31 15:15:16.208986 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.208879    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209072    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home" (OuterVolumeSpecName: "home") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209086    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home" (OuterVolumeSpecName: "home") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209196    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209216    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209230    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209278    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209291    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.209477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.209314    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.211496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211460    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm" (OuterVolumeSpecName: "dshm") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.211496 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211475    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:15:16.211648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211502    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm" (OuterVolumeSpecName: "dshm") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:15:16.211648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211553    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:15:16.211648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211564    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v" (OuterVolumeSpecName: "kube-api-access-wd55v") pod "d220041a-133c-4b7b-aaae-847b918fc809" (UID: "d220041a-133c-4b7b-aaae-847b918fc809"). InnerVolumeSpecName "kube-api-access-wd55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:15:16.211648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.211643    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x" (OuterVolumeSpecName: "kube-api-access-g4l5x") pod "9b3285b8-2799-4fec-80e0-fa0723e8042d" (UID: "9b3285b8-2799-4fec-80e0-fa0723e8042d"). InnerVolumeSpecName "kube-api-access-g4l5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:15:16.310318 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310277    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b3285b8-2799-4fec-80e0-fa0723e8042d-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310318 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310310    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d220041a-133c-4b7b-aaae-847b918fc809-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310318 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310318    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310318 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310328    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d220041a-133c-4b7b-aaae-847b918fc809-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310614 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310338    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4l5x\" (UniqueName: \"kubernetes.io/projected/9b3285b8-2799-4fec-80e0-fa0723e8042d-kube-api-access-g4l5x\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310614 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310347    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b3285b8-2799-4fec-80e0-fa0723e8042d-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.310614 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.310356    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd55v\" (UniqueName: \"kubernetes.io/projected/d220041a-133c-4b7b-aaae-847b918fc809-kube-api-access-wd55v\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:15:16.849045 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.849012    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"
Jul 31 15:15:16.849541 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.849015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r" event={"ID":"9b3285b8-2799-4fec-80e0-fa0723e8042d","Type":"ContainerDied","Data":"221fe9a3f35cd3d4c08905299e1103f25c44fddcd5a9f5ce62a84cc39b279d73"}
Jul 31 15:15:16.849541 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.849147    2557 scope.go:117] "RemoveContainer" containerID="b1382c8d6157041c043ccae38592e032972e93c64a3e010d54e1c2c420745490"
Jul 31 15:15:16.850556 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.850423    2557 generic.go:356] "Generic (PLEG): container finished" podID="d220041a-133c-4b7b-aaae-847b918fc809" containerID="ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2" exitCode=0
Jul 31 15:15:16.850556 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.850462    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"
Jul 31 15:15:16.850556 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.850508    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" event={"ID":"d220041a-133c-4b7b-aaae-847b918fc809","Type":"ContainerDied","Data":"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"}
Jul 31 15:15:16.850700 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.850564    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc" event={"ID":"d220041a-133c-4b7b-aaae-847b918fc809","Type":"ContainerDied","Data":"4205580b8d3244e98452dbda5a5c28026812058dc8eeaf0d8255442943b3ab29"}
Jul 31 15:15:16.860104 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.860085    2557 scope.go:117] "RemoveContainer" containerID="ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"
Jul 31 15:15:16.868547 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.868510    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:15:16.874443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.874407    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-cp77r"]
Jul 31 15:15:16.875713 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.875693    2557 scope.go:117] "RemoveContainer" containerID="ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"
Jul 31 15:15:16.876035 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:15:16.876015    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2\": container with ID starting with ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2 not found: ID does not exist" containerID="ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"
Jul 31 15:15:16.876082 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.876044    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2"} err="failed to get container status \"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2\": rpc error: code = NotFound desc = could not find container \"ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2\": container with ID starting with ff2f07457cc38a5fa3c5bcacd8107957175f30cbe703e2ec5c369b85cf0d56f2 not found: ID does not exist"
Jul 31 15:15:16.885444 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.885419    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:15:16.886856 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:16.886833    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-dnddc"]
Jul 31 15:15:18.247418 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.247383    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3285b8-2799-4fec-80e0-fa0723e8042d" path="/var/lib/kubelet/pods/9b3285b8-2799-4fec-80e0-fa0723e8042d/volumes"
Jul 31 15:15:18.247815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.247775    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d220041a-133c-4b7b-aaae-847b918fc809" path="/var/lib/kubelet/pods/d220041a-133c-4b7b-aaae-847b918fc809/volumes"
Jul 31 15:15:18.613657 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613578    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:15:18.613916 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613904    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="main"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613917    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="main"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613926    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="storage-initializer"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613933    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="storage-initializer"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613943    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3285b8-2799-4fec-80e0-fa0723e8042d" containerName="main"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613950    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3285b8-2799-4fec-80e0-fa0723e8042d" containerName="main"
Jul 31 15:15:18.613957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613959    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d220041a-133c-4b7b-aaae-847b918fc809" containerName="main"
Jul 31 15:15:18.614146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.613965    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d220041a-133c-4b7b-aaae-847b918fc809" containerName="main"
Jul 31 15:15:18.614146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.614028    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3285b8-2799-4fec-80e0-fa0723e8042d" containerName="main"
Jul 31 15:15:18.614146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.614035    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d220041a-133c-4b7b-aaae-847b918fc809" containerName="main"
Jul 31 15:15:18.614146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.614042    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4700d4c3-4ea4-4f27-97cf-fc9184cea7aa" containerName="main"
Jul 31 15:15:18.618533 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.618509    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.622045 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.622013    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-gfzr2\""
Jul 31 15:15:18.622045 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.622037    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Jul 31 15:15:18.622273 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.622013    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Jul 31 15:15:18.622273 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.622012    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Jul 31 15:15:18.627307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.627270    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:15:18.733214 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733170    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v8f\" (UniqueName: \"kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.733429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.733429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733354    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.733429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733388    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.733429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733409    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.733601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.733491    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834324 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834281    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834324 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834359    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834421    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834466    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v8f\" (UniqueName: \"kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834496    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834811 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834700    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834811 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834748    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.834811 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.834799    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.836741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.836698    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.836968 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.836954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.844923 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.844900    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v8f\" (UniqueName: \"kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f\") pod \"test-div-v2-kserve-5f6fdb6b77-5f9mm\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:18.929352 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:18.929312    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:19.066688 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:19.066664    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:15:19.069253 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:15:19.069210    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4bd968_abf0_4fe4_a968_44876665ad83.slice/crio-a4a5a14d1fd0bcd50970eef72d5617bce0efa6701bfdacc8bdcbbc8e1e3951b6 WatchSource:0}: Error finding container a4a5a14d1fd0bcd50970eef72d5617bce0efa6701bfdacc8bdcbbc8e1e3951b6: Status 404 returned error can't find the container with id a4a5a14d1fd0bcd50970eef72d5617bce0efa6701bfdacc8bdcbbc8e1e3951b6
Jul 31 15:15:19.869212 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:19.869178    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" event={"ID":"7e4bd968-abf0-4fe4-a968-44876665ad83","Type":"ContainerStarted","Data":"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"}
Jul 31 15:15:19.869905 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:19.869226    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" event={"ID":"7e4bd968-abf0-4fe4-a968-44876665ad83","Type":"ContainerStarted","Data":"a4a5a14d1fd0bcd50970eef72d5617bce0efa6701bfdacc8bdcbbc8e1e3951b6"}
Jul 31 15:15:19.892513 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:19.892459    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" podStartSLOduration=1.8924427000000001 podStartE2EDuration="1.8924427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:19.892349037 +0000 UTC m=+1642.050600529" watchObservedRunningTime="2026-07-31 15:15:19.8924427 +0000 UTC m=+1642.050694226"
Jul 31 15:15:19.930431 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:19.930393    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:28.929949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:28.929905    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:28.942431 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:28.942405    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:29.928142 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:29.928114    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:15:46.129290 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.129257    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:15:46.132694 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.132677    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.135649 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.135624    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:15:46.135649 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.135634    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 31 15:15:46.136417 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.136386    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 31 15:15:46.136531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.136429    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 31 15:15:46.136531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.136486    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-h48tb\""
Jul 31 15:15:46.136531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.136524    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 31 15:15:46.144673 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.144650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:15:46.273634 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273597    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.273634 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273635    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.273865 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273654    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.273865 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273740    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.273865 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273774    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.273865 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273824    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.274024 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273872    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.274024 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.273888    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lqb\" (UniqueName: \"kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375131 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375098    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375131 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375129    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375180    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375216    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375272    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375332    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375354    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lqb\" (UniqueName: \"kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375703 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375570    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375703 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375611    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375795 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375718    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.375843 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375828    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.376016 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.375993    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.377636 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.377614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.377788 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.377770    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.385714 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.385657    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lqb\" (UniqueName: \"kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb\") pod \"tokenizer-migration-path-test-kserve-68cffc78cd-tprgh\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.444405 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.444369    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:46.586596 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:46.586393    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:15:46.588961 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:15:46.588917    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160792ef_7da0_4b35_a441_807e4837a08e.slice/crio-925852f427beec55af1eba304e30b548d2eca87a2aaea3180e33ef572053924d WatchSource:0}: Error finding container 925852f427beec55af1eba304e30b548d2eca87a2aaea3180e33ef572053924d: Status 404 returned error can't find the container with id 925852f427beec55af1eba304e30b548d2eca87a2aaea3180e33ef572053924d
Jul 31 15:15:47.000080 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:47.000047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerStarted","Data":"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"}
Jul 31 15:15:47.000271 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:47.000101    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerStarted","Data":"925852f427beec55af1eba304e30b548d2eca87a2aaea3180e33ef572053924d"}
Jul 31 15:15:49.010535 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:49.010499    2557 generic.go:356] "Generic (PLEG): container finished" podID="160792ef-7da0-4b35-a441-807e4837a08e" containerID="5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa" exitCode=0
Jul 31 15:15:49.010921 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:49.010575    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerDied","Data":"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"}
Jul 31 15:15:50.017736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:50.017697    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerStarted","Data":"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"}
Jul 31 15:15:50.040639 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:50.040587    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" podStartSLOduration=1.809386141 podStartE2EDuration="4.040571055s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.231184914s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:50.040275016 +0000 UTC m=+1672.198526499" watchObservedRunningTime="2026-07-31 15:15:50.040571055 +0000 UTC m=+1672.198822546"
Jul 31 15:15:50.445494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:50.445458    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:56.445193 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:56.445150    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:56.457691 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:56.457661    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:15:57.060526 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:15:57.060499    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:16:23.457436 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.457346    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:16:23.467699 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:16:23.467654    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:16:23.467699 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.467699 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.467699 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174" cmd=["/bin/sleep","15"]
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.467725    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" message="" err=<
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:16:23.467746    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="main" containerID="cri-o://3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"
Jul 31 15:16:23.468043 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.467795    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="main" containerID="cri-o://3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174" gracePeriod=60
Jul 31 15:16:23.824511 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.824483    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:16:23.917367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917335    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917375    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917429    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2lqb\" (UniqueName: \"kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917486    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917526    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917549    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917863 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917639    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917863 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917664    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location\") pod \"160792ef-7da0-4b35-a441-807e4837a08e\" (UID: \"160792ef-7da0-4b35-a441-807e4837a08e\") "
Jul 31 15:16:23.917863 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917661    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache" (OuterVolumeSpecName: "model-cache") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:23.917863 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917727    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home" (OuterVolumeSpecName: "home") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:23.918081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.917873    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:23.918081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.918034    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:23.918081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.918056    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:23.918081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.918070    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:23.918081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.918065    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:23.919951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.919928    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm" (OuterVolumeSpecName: "dshm") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:23.920117 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.920099    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb" (OuterVolumeSpecName: "kube-api-access-k2lqb") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "kube-api-access-k2lqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:23.920172 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.920126    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:16:23.939401 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:23.939372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "160792ef-7da0-4b35-a441-807e4837a08e" (UID: "160792ef-7da0-4b35-a441-807e4837a08e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:24.018589 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.018505    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.018589 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.018531    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/160792ef-7da0-4b35-a441-807e4837a08e-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.018589 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.018543    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/160792ef-7da0-4b35-a441-807e4837a08e-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.018589 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.018554    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lqb\" (UniqueName: \"kubernetes.io/projected/160792ef-7da0-4b35-a441-807e4837a08e-kube-api-access-k2lqb\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.018589 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.018563    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/160792ef-7da0-4b35-a441-807e4837a08e-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:24.187886 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.187849    2557 generic.go:356] "Generic (PLEG): container finished" podID="160792ef-7da0-4b35-a441-807e4837a08e" containerID="3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174" exitCode=0
Jul 31 15:16:24.188117 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.187937    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"
Jul 31 15:16:24.188117 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.187950    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerDied","Data":"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"}
Jul 31 15:16:24.188117 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.188047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh" event={"ID":"160792ef-7da0-4b35-a441-807e4837a08e","Type":"ContainerDied","Data":"925852f427beec55af1eba304e30b548d2eca87a2aaea3180e33ef572053924d"}
Jul 31 15:16:24.188117 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.188072    2557 scope.go:117] "RemoveContainer" containerID="3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"
Jul 31 15:16:24.197074 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.197054    2557 scope.go:117] "RemoveContainer" containerID="5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"
Jul 31 15:16:24.205367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.205350    2557 scope.go:117] "RemoveContainer" containerID="3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"
Jul 31 15:16:24.205662 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:16:24.205642    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174\": container with ID starting with 3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174 not found: ID does not exist" containerID="3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"
Jul 31 15:16:24.205716 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.205672    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174"} err="failed to get container status \"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174\": rpc error: code = NotFound desc = could not find container \"3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174\": container with ID starting with 3c1e20aa738a2a3e1d4779b4aa1c61b46fccc3a2be34e43c2a2cd432a6803174 not found: ID does not exist"
Jul 31 15:16:24.205716 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.205690    2557 scope.go:117] "RemoveContainer" containerID="5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"
Jul 31 15:16:24.205917 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:16:24.205899    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa\": container with ID starting with 5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa not found: ID does not exist" containerID="5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"
Jul 31 15:16:24.205974 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.205926    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa"} err="failed to get container status \"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa\": rpc error: code = NotFound desc = could not find container \"5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa\": container with ID starting with 5c8adc2f6ef09180452a9908e73fda3defea9a2a80ebd148983d0b05842326fa not found: ID does not exist"
Jul 31 15:16:24.211214 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.211193    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:16:24.215530 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.215510    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-68cffc78cd-tprgh"]
Jul 31 15:16:24.247780 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:24.247755    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160792ef-7da0-4b35-a441-807e4837a08e" path="/var/lib/kubelet/pods/160792ef-7da0-4b35-a441-807e4837a08e/volumes"
Jul 31 15:16:30.818189 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:30.818146    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:16:45.837503 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:45.837434    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerName="main" containerID="cri-o://23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222" gracePeriod=45
Jul 31 15:16:45.917561 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:45.917517    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 31 15:16:46.100142 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.100075    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:16:46.225472 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225433    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225680 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225509    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225680 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225569    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225680 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225600    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225680 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225647    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225680 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225676    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5v8f\" (UniqueName: \"kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f\") pod \"7e4bd968-abf0-4fe4-a968-44876665ad83\" (UID: \"7e4bd968-abf0-4fe4-a968-44876665ad83\") "
Jul 31 15:16:46.225937 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225804    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.225937 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225817    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home" (OuterVolumeSpecName: "home") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.225937 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225834    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.226093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225971    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.226093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.225989    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.226093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.226001    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.227940 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.227916    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f" (OuterVolumeSpecName: "kube-api-access-b5v8f") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "kube-api-access-b5v8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:46.228298 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.228278    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm" (OuterVolumeSpecName: "dshm") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:46.228359 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.228301    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e4bd968-abf0-4fe4-a968-44876665ad83" (UID: "7e4bd968-abf0-4fe4-a968-44876665ad83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:16:46.290570 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.290531    2557 generic.go:356] "Generic (PLEG): container finished" podID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerID="23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222" exitCode=0
Jul 31 15:16:46.290751 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.290595    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"
Jul 31 15:16:46.290751 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.290618    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" event={"ID":"7e4bd968-abf0-4fe4-a968-44876665ad83","Type":"ContainerDied","Data":"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"}
Jul 31 15:16:46.290751 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.290680    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm" event={"ID":"7e4bd968-abf0-4fe4-a968-44876665ad83","Type":"ContainerDied","Data":"a4a5a14d1fd0bcd50970eef72d5617bce0efa6701bfdacc8bdcbbc8e1e3951b6"}
Jul 31 15:16:46.290751 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.290710    2557 scope.go:117] "RemoveContainer" containerID="23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"
Jul 31 15:16:46.299649 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.299630    2557 scope.go:117] "RemoveContainer" containerID="23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"
Jul 31 15:16:46.299934 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:16:46.299915    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222\": container with ID starting with 23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222 not found: ID does not exist" containerID="23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"
Jul 31 15:16:46.299999 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.299943    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222"} err="failed to get container status \"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222\": rpc error: code = NotFound desc = could not find container \"23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222\": container with ID starting with 23fe113340242448fa8ce8ae04ef29fe178e0f729f691f54b2cabb0978290222 not found: ID does not exist"
Jul 31 15:16:46.307052 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.307027    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:16:46.310545 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.310521    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-5f9mm"]
Jul 31 15:16:46.327355 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.327321    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e4bd968-abf0-4fe4-a968-44876665ad83-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.327355 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.327346    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5v8f\" (UniqueName: \"kubernetes.io/projected/7e4bd968-abf0-4fe4-a968-44876665ad83-kube-api-access-b5v8f\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:46.327355 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:46.327357    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e4bd968-abf0-4fe4-a968-44876665ad83-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:16:48.247714 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:16:48.247680    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" path="/var/lib/kubelet/pods/7e4bd968-abf0-4fe4-a968-44876665ad83/volumes"
Jul 31 15:17:58.296317 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:17:58.296279    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:17:58.300977 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:17:58.300953    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:18:06.405748 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.405674    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406036    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerName="main"
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406048    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerName="main"
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406057    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="main"
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406062    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="main"
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406075    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="storage-initializer"
Jul 31 15:18:06.406093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406082    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="storage-initializer"
Jul 31 15:18:06.406345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406150    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e4bd968-abf0-4fe4-a968-44876665ad83" containerName="main"
Jul 31 15:18:06.406345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.406163    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="160792ef-7da0-4b35-a441-807e4837a08e" containerName="main"
Jul 31 15:18:06.409051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.409034    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.412624 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.412602    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-n6qqx\""
Jul 31 15:18:06.412754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.412605    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Jul 31 15:18:06.412754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.412648    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Jul 31 15:18:06.412754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.412637    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Jul 31 15:18:06.422315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.422292    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:18:06.541796 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.541759    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.542001 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.541816    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.542001 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.541875    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.542001 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.541916    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.542141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.542027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.542141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.542073    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnqb\" (UniqueName: \"kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642512 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642470    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642567    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642611    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnqb\" (UniqueName: \"kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642639    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642679    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.642722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.642705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.643041 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.643014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.643107 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.643041    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.643151 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.643129    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.644808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.644789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.645157 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.645137    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.651270 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.651229    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnqb\" (UniqueName: \"kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb\") pod \"decom-v1-kserve-6cb797fb75-7kxkf\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.718961 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.718917    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:06.855426 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:06.855233    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:18:06.858217 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:18:06.858183    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edbb763_865a_4dfb_a077_411521cb4c36.slice/crio-82c2475bfd41ac34cd16ad5239092887f75a369363004192338d33f50a4f8ad7 WatchSource:0}: Error finding container 82c2475bfd41ac34cd16ad5239092887f75a369363004192338d33f50a4f8ad7: Status 404 returned error can't find the container with id 82c2475bfd41ac34cd16ad5239092887f75a369363004192338d33f50a4f8ad7
Jul 31 15:18:07.659834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:07.659794    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" event={"ID":"6edbb763-865a-4dfb-a077-411521cb4c36","Type":"ContainerStarted","Data":"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"}
Jul 31 15:18:07.660288 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:07.659855    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" event={"ID":"6edbb763-865a-4dfb-a077-411521cb4c36","Type":"ContainerStarted","Data":"82c2475bfd41ac34cd16ad5239092887f75a369363004192338d33f50a4f8ad7"}
Jul 31 15:18:07.684646 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:07.684593    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" podStartSLOduration=1.6845762130000002 podStartE2EDuration="1.684576213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:07.682490566 +0000 UTC m=+1809.840742056" watchObservedRunningTime="2026-07-31 15:18:07.684576213 +0000 UTC m=+1809.842827703"
Jul 31 15:18:07.719784 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:07.719753    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:16.719631 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:16.719596    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:16.731952 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:16.731924    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:17.717438 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:17.717408    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:18:28.236513 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.236480    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:18:28.239667 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.239651    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.242792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.242769    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Jul 31 15:18:28.252395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252368    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.252520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252407    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.252520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252431    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.252520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252503    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.252630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252592    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2bm\" (UniqueName: \"kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.252630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.252620    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.256299 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.256276    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:18:28.353598 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.353598 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353602    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.353821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353629    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.353821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.353821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2bm\" (UniqueName: \"kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.354040 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.353881    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.354133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.354102    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.354172 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.354135    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.354218 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.354188    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.355983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.355955    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.356369 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.356350    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.363180 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.363154    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2bm\" (UniqueName: \"kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm\") pod \"decom-v2-kserve-77f867d787-td69c\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.550409 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.550305    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:28.688640 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.688519    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:18:28.691003 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:18:28.690974    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bcb2c3_df1e_4a98_987c_c19bd9203965.slice/crio-badd1d6d0cf890d88a7853f89d540bcf2ccf12133133890457872fb373ba5043 WatchSource:0}: Error finding container badd1d6d0cf890d88a7853f89d540bcf2ccf12133133890457872fb373ba5043: Status 404 returned error can't find the container with id badd1d6d0cf890d88a7853f89d540bcf2ccf12133133890457872fb373ba5043
Jul 31 15:18:28.758328 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.758286    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" event={"ID":"b4bcb2c3-df1e-4a98-987c-c19bd9203965","Type":"ContainerStarted","Data":"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"}
Jul 31 15:18:28.758489 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.758355    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" event={"ID":"b4bcb2c3-df1e-4a98-987c-c19bd9203965","Type":"ContainerStarted","Data":"badd1d6d0cf890d88a7853f89d540bcf2ccf12133133890457872fb373ba5043"}
Jul 31 15:18:28.780372 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:28.780320    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" podStartSLOduration=0.780302616 podStartE2EDuration="780.302616ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:28.779216509 +0000 UTC m=+1830.937468012" watchObservedRunningTime="2026-07-31 15:18:28.780302616 +0000 UTC m=+1830.938554148"
Jul 31 15:18:29.551203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:29.551165    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:38.550513 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:38.550475    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:38.563136 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:38.563107    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:18:38.819288 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:18:38.819192    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:19:15.532006 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:15.531974    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:19:30.551803 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.551681    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" podUID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" containerName="main" containerID="cri-o://06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6" gracePeriod=45
Jul 31 15:19:30.806787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.806726    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:19:30.898256 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898216    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898286    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898312    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m2bm\" (UniqueName: \"kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898355    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898397    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898652 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898436    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm\") pod \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\" (UID: \"b4bcb2c3-df1e-4a98-987c-c19bd9203965\") "
Jul 31 15:19:30.898652 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898466    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home" (OuterVolumeSpecName: "home") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:30.898652 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898590    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:30.898803 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898664    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:30.898803 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898724    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:30.898803 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898738    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:30.898803 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.898747    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:30.900570 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.900544    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm" (OuterVolumeSpecName: "dshm") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:30.900671 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.900602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm" (OuterVolumeSpecName: "kube-api-access-5m2bm") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "kube-api-access-5m2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:19:30.900671 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.900602    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4bcb2c3-df1e-4a98-987c-c19bd9203965" (UID: "b4bcb2c3-df1e-4a98-987c-c19bd9203965"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:19:30.999976 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.999936    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bcb2c3-df1e-4a98-987c-c19bd9203965-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:30.999976 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.999965    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m2bm\" (UniqueName: \"kubernetes.io/projected/b4bcb2c3-df1e-4a98-987c-c19bd9203965-kube-api-access-5m2bm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:30.999976 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:30.999975    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bcb2c3-df1e-4a98-987c-c19bd9203965-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:31.054657 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.054624    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" containerID="06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6" exitCode=0
Jul 31 15:19:31.054657 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.054662    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" event={"ID":"b4bcb2c3-df1e-4a98-987c-c19bd9203965","Type":"ContainerDied","Data":"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"}
Jul 31 15:19:31.054864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.054666    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"
Jul 31 15:19:31.054864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.054711    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c" event={"ID":"b4bcb2c3-df1e-4a98-987c-c19bd9203965","Type":"ContainerDied","Data":"badd1d6d0cf890d88a7853f89d540bcf2ccf12133133890457872fb373ba5043"}
Jul 31 15:19:31.054864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.054734    2557 scope.go:117] "RemoveContainer" containerID="06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"
Jul 31 15:19:31.068858 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.068839    2557 scope.go:117] "RemoveContainer" containerID="06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"
Jul 31 15:19:31.069108 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:19:31.069089    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6\": container with ID starting with 06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6 not found: ID does not exist" containerID="06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"
Jul 31 15:19:31.069163 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.069115    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6"} err="failed to get container status \"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6\": rpc error: code = NotFound desc = could not find container \"06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6\": container with ID starting with 06dd29f20878d06d6f059475b74913a5d3b49f4a37ddcccb3c8e63f760fe95e6 not found: ID does not exist"
Jul 31 15:19:31.086711 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.086682    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:19:31.088114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:31.088096    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-td69c"]
Jul 31 15:19:32.248359 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:32.248324    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" path="/var/lib/kubelet/pods/b4bcb2c3-df1e-4a98-987c-c19bd9203965/volumes"
Jul 31 15:19:36.516393 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:36.516355    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:19:44.598468 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.598433    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:19:44.598857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.598830    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" containerName="main"
Jul 31 15:19:44.598857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.598843    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" containerName="main"
Jul 31 15:19:44.598952 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.598923    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4bcb2c3-df1e-4a98-987c-c19bd9203965" containerName="main"
Jul 31 15:19:44.603153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.603126    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.606708 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.606684    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Jul 31 15:19:44.606851 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.606706    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Jul 31 15:19:44.606851 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.606757    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-zww8c\""
Jul 31 15:19:44.606851 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.606687    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Jul 31 15:19:44.611981 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.611954    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.723253    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.723757    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.723823    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.723863    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.723915    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.727258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.724027    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmst\" (UniqueName: \"kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824647 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824602    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824837 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824700    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmst\" (UniqueName: \"kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824837 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824738    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824837 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824758    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824837 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824782    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.824837 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.824802    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.825123 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.825033    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.825203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.825179    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.825294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.825200    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.827162 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.827140    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.827260 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.827228    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.834507 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.834460    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmst\" (UniqueName: \"kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst\") pod \"leave-v2-kserve-556b4dddf9-pzwf6\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:44.914307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:44.914213    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:45.060119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:45.060086    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:19:45.062428 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:19:45.062385    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744b876d_56f4_4d22_8fd0_ac95ebe96581.slice/crio-f9c82128116d104b858b9bb2851b92aa0d501824804fb245d535b8f63e2b536f WatchSource:0}: Error finding container f9c82128116d104b858b9bb2851b92aa0d501824804fb245d535b8f63e2b536f: Status 404 returned error can't find the container with id f9c82128116d104b858b9bb2851b92aa0d501824804fb245d535b8f63e2b536f
Jul 31 15:19:45.122890 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:45.122860    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" event={"ID":"744b876d-56f4-4d22-8fd0-ac95ebe96581","Type":"ContainerStarted","Data":"f9c82128116d104b858b9bb2851b92aa0d501824804fb245d535b8f63e2b536f"}
Jul 31 15:19:46.133313 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:46.133272    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" event={"ID":"744b876d-56f4-4d22-8fd0-ac95ebe96581","Type":"ContainerStarted","Data":"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"}
Jul 31 15:19:46.156539 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:46.156485    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" podStartSLOduration=2.15646753 podStartE2EDuration="2.15646753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:19:46.155359348 +0000 UTC m=+1908.313610840" watchObservedRunningTime="2026-07-31 15:19:46.15646753 +0000 UTC m=+1908.314719020"
Jul 31 15:19:46.914819 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:46.914777    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:51.535520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.535467    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" containerName="main" containerID="cri-o://8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb" gracePeriod=45
Jul 31 15:19:51.706781 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.706744    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jul 31 15:19:51.804949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.804926    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:19:51.891666 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891637    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.891864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891679    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.891864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891699    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.891864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891729    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.891864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891765    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krnqb\" (UniqueName: \"kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.891864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891845    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir\") pod \"6edbb763-865a-4dfb-a077-411521cb4c36\" (UID: \"6edbb763-865a-4dfb-a077-411521cb4c36\") "
Jul 31 15:19:51.892170 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891956    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home" (OuterVolumeSpecName: "home") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:51.892170 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.891972    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache" (OuterVolumeSpecName: "model-cache") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:51.892293 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.892168    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:51.892293 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.892191    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:51.892293 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.892209    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:51.893830 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.893807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:19:51.893932 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.893831    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb" (OuterVolumeSpecName: "kube-api-access-krnqb") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "kube-api-access-krnqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:19:51.893932 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.893880    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm" (OuterVolumeSpecName: "dshm") pod "6edbb763-865a-4dfb-a077-411521cb4c36" (UID: "6edbb763-865a-4dfb-a077-411521cb4c36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:19:51.992763 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.992730    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:51.992763 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.992758    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnqb\" (UniqueName: \"kubernetes.io/projected/6edbb763-865a-4dfb-a077-411521cb4c36-kube-api-access-krnqb\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:51.992763 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.992769    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6edbb763-865a-4dfb-a077-411521cb4c36-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:51.993040 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:51.992779    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6edbb763-865a-4dfb-a077-411521cb4c36-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:19:52.162918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.162826    2557 generic.go:356] "Generic (PLEG): container finished" podID="6edbb763-865a-4dfb-a077-411521cb4c36" containerID="8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb" exitCode=0
Jul 31 15:19:52.162918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.162873    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" event={"ID":"6edbb763-865a-4dfb-a077-411521cb4c36","Type":"ContainerDied","Data":"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"}
Jul 31 15:19:52.162918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.162917    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf" event={"ID":"6edbb763-865a-4dfb-a077-411521cb4c36","Type":"ContainerDied","Data":"82c2475bfd41ac34cd16ad5239092887f75a369363004192338d33f50a4f8ad7"}
Jul 31 15:19:52.163164 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.162938    2557 scope.go:117] "RemoveContainer" containerID="8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"
Jul 31 15:19:52.163164 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.162877    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"
Jul 31 15:19:52.171969 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.171950    2557 scope.go:117] "RemoveContainer" containerID="8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"
Jul 31 15:19:52.172308 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:19:52.172283    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb\": container with ID starting with 8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb not found: ID does not exist" containerID="8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"
Jul 31 15:19:52.172412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.172317    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb"} err="failed to get container status \"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb\": rpc error: code = NotFound desc = could not find container \"8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb\": container with ID starting with 8084fd81274c53b4ee00b8eb3c35a751bf1d16410a4d539aa78acae0434030cb not found: ID does not exist"
Jul 31 15:19:52.189285 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.189231    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:19:52.194047 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.194024    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-7kxkf"]
Jul 31 15:19:52.247628 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:52.247587    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" path="/var/lib/kubelet/pods/6edbb763-865a-4dfb-a077-411521cb4c36/volumes"
Jul 31 15:19:54.915000 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:54.914961    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:54.927627 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:54.927603    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:55.189622 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:55.189590    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:19:59.295869 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.295833    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:19:59.296477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.296456    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" containerName="main"
Jul 31 15:19:59.296545 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.296480    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" containerName="main"
Jul 31 15:19:59.296592 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.296575    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6edbb763-865a-4dfb-a077-411521cb4c36" containerName="main"
Jul 31 15:19:59.301017 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.300999    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.304718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.304696    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 31 15:19:59.304864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.304840    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-qptkq\""
Jul 31 15:19:59.304864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.304850    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 31 15:19:59.304972 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.304888    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 31 15:19:59.312560 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.312537    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:19:59.460742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460704    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.460919 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460760    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.460919 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460838    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.460919 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460866    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nph5s\" (UniqueName: \"kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.460919 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460904    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.461084 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.460940    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562006 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.561916    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562006 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.561955    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nph5s\" (UniqueName: \"kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562006 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562001    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562043    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562110    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562510 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562497    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562566 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562545    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.562603 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.562548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.564278 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.564257    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.564560 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.564543    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.579521 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.579487    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nph5s\" (UniqueName: \"kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.611499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.611465    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:19:59.760216 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:19:59.760112    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:19:59.762938 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:19:59.762915    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdcced9_f8d3_4713_910e_b4a60b610188.slice/crio-e27984847d7bc7e030deee93cc81d094d3edf8d31f025e5eb2e4d1b6b4db72dc WatchSource:0}: Error finding container e27984847d7bc7e030deee93cc81d094d3edf8d31f025e5eb2e4d1b6b4db72dc: Status 404 returned error can't find the container with id e27984847d7bc7e030deee93cc81d094d3edf8d31f025e5eb2e4d1b6b4db72dc
Jul 31 15:20:00.205988 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:00.205956    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" event={"ID":"dfdcced9-f8d3-4713-910e-b4a60b610188","Type":"ContainerStarted","Data":"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"}
Jul 31 15:20:00.206154 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:00.206015    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" event={"ID":"dfdcced9-f8d3-4713-910e-b4a60b610188","Type":"ContainerStarted","Data":"e27984847d7bc7e030deee93cc81d094d3edf8d31f025e5eb2e4d1b6b4db72dc"}
Jul 31 15:20:00.226524 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:00.226468    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" podStartSLOduration=1.226450961 podStartE2EDuration="1.226450961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:00.225579407 +0000 UTC m=+1922.383830908" watchObservedRunningTime="2026-07-31 15:20:00.226450961 +0000 UTC m=+1922.384702451"
Jul 31 15:20:00.611834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:00.611747    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:20:09.612623 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:09.612580    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:20:09.625423 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:09.625386    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:20:10.265220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:10.265191    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:20:34.454199 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:34.454164    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:20:49.473795 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.473721    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" podUID="744b876d-56f4-4d22-8fd0-ac95ebe96581" containerName="main" containerID="cri-o://ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7" gracePeriod=45
Jul 31 15:20:49.719331 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.719301    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:20:49.825571 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825477    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825571 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825531    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825571 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825555    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825578    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825629    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hmst\" (UniqueName: \"kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825674    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home\") pod \"744b876d-56f4-4d22-8fd0-ac95ebe96581\" (UID: \"744b876d-56f4-4d22-8fd0-ac95ebe96581\") "
Jul 31 15:20:49.825832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825807    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.826047 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache" (OuterVolumeSpecName: "model-cache") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.826047 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.825994    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home" (OuterVolumeSpecName: "home") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.826128 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.826053    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.826128 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.826073    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.827823 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.827798    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm" (OuterVolumeSpecName: "dshm") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:20:49.827949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.827825    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst" (OuterVolumeSpecName: "kube-api-access-7hmst") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "kube-api-access-7hmst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:20:49.827949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.827836    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "744b876d-56f4-4d22-8fd0-ac95ebe96581" (UID: "744b876d-56f4-4d22-8fd0-ac95ebe96581"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:20:49.926993 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.926958    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/744b876d-56f4-4d22-8fd0-ac95ebe96581-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.926993 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.926985    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.926993 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.926996    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmst\" (UniqueName: \"kubernetes.io/projected/744b876d-56f4-4d22-8fd0-ac95ebe96581-kube-api-access-7hmst\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:49.926993 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:49.927005    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/744b876d-56f4-4d22-8fd0-ac95ebe96581-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:20:50.445257 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.445209    2557 generic.go:356] "Generic (PLEG): container finished" podID="744b876d-56f4-4d22-8fd0-ac95ebe96581" containerID="ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7" exitCode=0
Jul 31 15:20:50.445440 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.445270    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"
Jul 31 15:20:50.445440 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.445269    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" event={"ID":"744b876d-56f4-4d22-8fd0-ac95ebe96581","Type":"ContainerDied","Data":"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"}
Jul 31 15:20:50.445440 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.445322    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6" event={"ID":"744b876d-56f4-4d22-8fd0-ac95ebe96581","Type":"ContainerDied","Data":"f9c82128116d104b858b9bb2851b92aa0d501824804fb245d535b8f63e2b536f"}
Jul 31 15:20:50.445440 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.445344    2557 scope.go:117] "RemoveContainer" containerID="ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"
Jul 31 15:20:50.453780 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.453739    2557 scope.go:117] "RemoveContainer" containerID="ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"
Jul 31 15:20:50.453999 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:20:50.453978    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7\": container with ID starting with ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7 not found: ID does not exist" containerID="ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"
Jul 31 15:20:50.454062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.454010    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7"} err="failed to get container status \"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7\": rpc error: code = NotFound desc = could not find container \"ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7\": container with ID starting with ce057b411e02d00440947bf24b9657543f2f3f8804d6cdc8ddedbda28a9dd5a7 not found: ID does not exist"
Jul 31 15:20:50.464649 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.464626    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:20:50.468687 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:50.468667    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-pzwf6"]
Jul 31 15:20:52.247346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:52.247304    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="744b876d-56f4-4d22-8fd0-ac95ebe96581" path="/var/lib/kubelet/pods/744b876d-56f4-4d22-8fd0-ac95ebe96581/volumes"
Jul 31 15:20:53.309300 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.309268    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:20:53.309667 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.309638    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="744b876d-56f4-4d22-8fd0-ac95ebe96581" containerName="main"
Jul 31 15:20:53.309667 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.309649    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="744b876d-56f4-4d22-8fd0-ac95ebe96581" containerName="main"
Jul 31 15:20:53.309743 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.309702    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="744b876d-56f4-4d22-8fd0-ac95ebe96581" containerName="main"
Jul 31 15:20:53.314402 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.314382    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.319563 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.319541    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-h2rbf\""
Jul 31 15:20:53.319686 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.319549    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Jul 31 15:20:53.319686 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.319598    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Jul 31 15:20:53.319686 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.319549    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Jul 31 15:20:53.329339 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.329317    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:20:53.355984 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.355958    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgw56\" (UniqueName: \"kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.356125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.356001    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.356125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.356081    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.356125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.356096    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.356228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.356139    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.356228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.356162    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457130 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457096    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgw56\" (UniqueName: \"kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457130 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457140    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457398 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457177    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457398 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457198    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457398 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457227    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457398 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457262    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457696 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457666    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457835 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457781    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.457835 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.457794    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.459585 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.459564    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.459969 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.459954    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.467308 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.467288    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgw56\" (UniqueName: \"kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56\") pod \"tri-v1-kserve-79f8f98bc5-2vh59\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.625046 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.624957    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:53.759349 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:53.759325    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:20:53.761909 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:20:53.761884    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372bae9f_9fa4_4cef_9588_37d72f07fa50.slice/crio-3c9a6d9d2f234cfcc3b92b2f33efb21d0208473828acdef8a7f8040f5e71db2d WatchSource:0}: Error finding container 3c9a6d9d2f234cfcc3b92b2f33efb21d0208473828acdef8a7f8040f5e71db2d: Status 404 returned error can't find the container with id 3c9a6d9d2f234cfcc3b92b2f33efb21d0208473828acdef8a7f8040f5e71db2d
Jul 31 15:20:54.466480 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:54.466430    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" event={"ID":"372bae9f-9fa4-4cef-9588-37d72f07fa50","Type":"ContainerStarted","Data":"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"}
Jul 31 15:20:54.466932 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:54.466506    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" event={"ID":"372bae9f-9fa4-4cef-9588-37d72f07fa50","Type":"ContainerStarted","Data":"3c9a6d9d2f234cfcc3b92b2f33efb21d0208473828acdef8a7f8040f5e71db2d"}
Jul 31 15:20:54.489125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:54.489078    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" podStartSLOduration=1.489063744 podStartE2EDuration="1.489063744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:54.488050604 +0000 UTC m=+1976.646302094" watchObservedRunningTime="2026-07-31 15:20:54.489063744 +0000 UTC m=+1976.647315233"
Jul 31 15:20:54.625384 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:54.625357    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:20:57.990386 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:57.990352    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:20:57.994094 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:57.994069    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:57.996591 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:57.996569    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Jul 31 15:20:58.004866 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.004844    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:20:58.097897 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.097860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dsbf\" (UniqueName: \"kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.098095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.097922    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.098095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.097974    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.098095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.098010    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.098095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.098026    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.098095 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.098079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199630 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199579    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dsbf\" (UniqueName: \"kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199652    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199706    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.199852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.199769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.200115 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.200090    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.200115 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.200104    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.200179 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.200149    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.201964 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.201943    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.202425 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.202411    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Jul 31 15:20:58.208841 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.208819    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dsbf\" (UniqueName: \"kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.212555 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.212532    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-7pwvj\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.305885 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.305791    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:20:58.441741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.441715    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:20:58.443414 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:20:58.443386    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83db231_7174_4417_8c78_02c8bb08eca1.slice/crio-992d931940bb1c69987f6a027fb1bbecb9340b5d6b22e088a2d9876427ae1c9a WatchSource:0}: Error finding container 992d931940bb1c69987f6a027fb1bbecb9340b5d6b22e088a2d9876427ae1c9a: Status 404 returned error can't find the container with id 992d931940bb1c69987f6a027fb1bbecb9340b5d6b22e088a2d9876427ae1c9a
Jul 31 15:20:58.486563 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:58.486526    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" event={"ID":"c83db231-7174-4417-8c78-02c8bb08eca1","Type":"ContainerStarted","Data":"992d931940bb1c69987f6a027fb1bbecb9340b5d6b22e088a2d9876427ae1c9a"}
Jul 31 15:20:59.492847 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:59.492809    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" event={"ID":"c83db231-7174-4417-8c78-02c8bb08eca1","Type":"ContainerStarted","Data":"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"}
Jul 31 15:20:59.517244 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:20:59.517196    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" podStartSLOduration=2.517179265 podStartE2EDuration="2.517179265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:59.515350019 +0000 UTC m=+1981.673601508" watchObservedRunningTime="2026-07-31 15:20:59.517179265 +0000 UTC m=+1981.675430756"
Jul 31 15:21:00.306463 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:00.306429    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:21:03.626188 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:03.626147    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:21:03.638353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:03.638324    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:21:04.527551 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:04.527524    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:21:08.306308 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:08.306271    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:21:08.318231 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:08.318204    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:21:08.545605 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:08.545579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:21:23.907929 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:23.907893    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:21:23.918899 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:21:23.918853    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:21:23.918899 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:23.918899 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:23.918899 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98" cmd=["/bin/sleep","15"]
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:23.918906    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" message="" err=<
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:21:23.918919    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main" containerID="cri-o://34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"
Jul 31 15:21:23.919246 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:23.918964    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main" containerID="cri-o://34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98" gracePeriod=60
Jul 31 15:21:24.273521 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.273496    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:21:24.453880 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.453880 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.453880 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453845    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.453880 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453867    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nph5s\" (UniqueName: \"kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.454219 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453921    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.454219 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.453978    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir\") pod \"dfdcced9-f8d3-4713-910e-b4a60b610188\" (UID: \"dfdcced9-f8d3-4713-910e-b4a60b610188\") "
Jul 31 15:21:24.454219 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.454147    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:24.454390 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.454288    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:24.454435 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.454399    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.454435 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.454415    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.454532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.454434    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home" (OuterVolumeSpecName: "home") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:24.456135 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.456109    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:21:24.456430 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.456408    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm" (OuterVolumeSpecName: "dshm") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:21:24.456514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.456425    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s" (OuterVolumeSpecName: "kube-api-access-nph5s") pod "dfdcced9-f8d3-4713-910e-b4a60b610188" (UID: "dfdcced9-f8d3-4713-910e-b4a60b610188"). InnerVolumeSpecName "kube-api-access-nph5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:21:24.555306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.555269    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.555306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.555298    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfdcced9-f8d3-4713-910e-b4a60b610188-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.555306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.555309    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nph5s\" (UniqueName: \"kubernetes.io/projected/dfdcced9-f8d3-4713-910e-b4a60b610188-kube-api-access-nph5s\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.555306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.555318    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfdcced9-f8d3-4713-910e-b4a60b610188-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:21:24.613337 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.613300    2557 generic.go:356] "Generic (PLEG): container finished" podID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerID="34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98" exitCode=0
Jul 31 15:21:24.613337 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.613338    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"
Jul 31 15:21:24.613552 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.613361    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" event={"ID":"dfdcced9-f8d3-4713-910e-b4a60b610188","Type":"ContainerDied","Data":"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"}
Jul 31 15:21:24.613552 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.613414    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" event={"ID":"dfdcced9-f8d3-4713-910e-b4a60b610188","Type":"ContainerDied","Data":"e27984847d7bc7e030deee93cc81d094d3edf8d31f025e5eb2e4d1b6b4db72dc"}
Jul 31 15:21:24.613552 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.613444    2557 scope.go:117] "RemoveContainer" containerID="34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"
Jul 31 15:21:24.622620 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.622598    2557 scope.go:117] "RemoveContainer" containerID="34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"
Jul 31 15:21:24.622904 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:21:24.622885    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98\": container with ID starting with 34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98 not found: ID does not exist" containerID="34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"
Jul 31 15:21:24.622957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.622915    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98"} err="failed to get container status \"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98\": rpc error: code = NotFound desc = could not find container \"34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98\": container with ID starting with 34a3455f76ea9ddb13141b283eaee0d5bac44833b535f242b20136d994e18d98 not found: ID does not exist"
Jul 31 15:21:24.634734 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.634709    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:21:24.638385 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:24.638363    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9"]
Jul 31 15:21:25.254591 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:25.254550    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995ngnz9" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:21:26.248046 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:21:26.248009    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" path="/var/lib/kubelet/pods/dfdcced9-f8d3-4713-910e-b4a60b610188/volumes"
Jul 31 15:22:19.454210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:19.450457    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:22:19.458595 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:19.458555    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:22:34.473152 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.473049    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerName="main" containerID="cri-o://613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f" gracePeriod=45
Jul 31 15:22:34.517536 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.517499    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Jul 31 15:22:34.721183 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.721154    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:22:34.772827 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.772733    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgw56\" (UniqueName: \"kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.772827 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.772821    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.773049 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.772854    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.773049 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.772876    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.773049 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.772917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.773049 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773014    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm\") pod \"372bae9f-9fa4-4cef-9588-37d72f07fa50\" (UID: \"372bae9f-9fa4-4cef-9588-37d72f07fa50\") "
Jul 31 15:22:34.773269 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773117    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.773269 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773153    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache" (OuterVolumeSpecName: "model-cache") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.773378 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773280    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home" (OuterVolumeSpecName: "home") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.773429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773404    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.773429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773418    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.773429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.773427    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.775022 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.774995    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:34.775345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.775316    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm" (OuterVolumeSpecName: "dshm") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:34.775345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.775320    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56" (OuterVolumeSpecName: "kube-api-access-mgw56") pod "372bae9f-9fa4-4cef-9588-37d72f07fa50" (UID: "372bae9f-9fa4-4cef-9588-37d72f07fa50"). InnerVolumeSpecName "kube-api-access-mgw56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:34.874133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.874094    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372bae9f-9fa4-4cef-9588-37d72f07fa50-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.874133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.874127    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372bae9f-9fa4-4cef-9588-37d72f07fa50-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.874133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.874135    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgw56\" (UniqueName: \"kubernetes.io/projected/372bae9f-9fa4-4cef-9588-37d72f07fa50-kube-api-access-mgw56\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:34.890602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.890571    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" podUID="c83db231-7174-4417-8c78-02c8bb08eca1" containerName="main" containerID="cri-o://77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183" gracePeriod=45
Jul 31 15:22:34.940982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.940954    2557 generic.go:356] "Generic (PLEG): container finished" podID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerID="613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f" exitCode=0
Jul 31 15:22:34.941090 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.941001    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"
Jul 31 15:22:34.941090 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.941034    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" event={"ID":"372bae9f-9fa4-4cef-9588-37d72f07fa50","Type":"ContainerDied","Data":"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"}
Jul 31 15:22:34.941090 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.941084    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59" event={"ID":"372bae9f-9fa4-4cef-9588-37d72f07fa50","Type":"ContainerDied","Data":"3c9a6d9d2f234cfcc3b92b2f33efb21d0208473828acdef8a7f8040f5e71db2d"}
Jul 31 15:22:34.941215 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.941106    2557 scope.go:117] "RemoveContainer" containerID="613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"
Jul 31 15:22:34.962862 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.962834    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:22:34.971537 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:34.971507    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-2vh59"]
Jul 31 15:22:35.001893 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.001869    2557 scope.go:117] "RemoveContainer" containerID="613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"
Jul 31 15:22:35.002032 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002014    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:22:35.002371 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:22:35.002272    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f\": container with ID starting with 613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f not found: ID does not exist" containerID="613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"
Jul 31 15:22:35.002371 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002307    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f"} err="failed to get container status \"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f\": rpc error: code = NotFound desc = could not find container \"613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f\": container with ID starting with 613cad12e7340584eb64371218a756b0649ee00f388b108006bfec900da9e81f not found: ID does not exist"
Jul 31 15:22:35.002477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002430    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main"
Jul 31 15:22:35.002477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002443    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main"
Jul 31 15:22:35.002477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002463    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerName="main"
Jul 31 15:22:35.002477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002468    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerName="main"
Jul 31 15:22:35.002599 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002522    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfdcced9-f8d3-4713-910e-b4a60b610188" containerName="main"
Jul 31 15:22:35.002599 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.002530    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" containerName="main"
Jul 31 15:22:35.005650 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.005631    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.018358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.018326    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Jul 31 15:22:35.018358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.018337    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-skk46\""
Jul 31 15:22:35.018597 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.018438    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Jul 31 15:22:35.018666 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.018622    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:22:35.018864 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.018847    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Jul 31 15:22:35.075951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.075917    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.075951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.075952    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.076171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.075987    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.076171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.076018    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.076171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.076054    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksb9\" (UniqueName: \"kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.076171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.076105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.122583 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.122551    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:22:35.177145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177116    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dsbf\" (UniqueName: \"kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177203    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177270    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177297    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177329    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177479 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177360    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache\") pod \"c83db231-7174-4417-8c78-02c8bb08eca1\" (UID: \"c83db231-7174-4417-8c78-02c8bb08eca1\") "
Jul 31 15:22:35.177519 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177478    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.177575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177535    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.177629 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177590    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.177688 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177665    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksb9\" (UniqueName: \"kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.177755 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177725    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.177821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.177776    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.178124 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.178097    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.178400 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.178378    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.178617 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.178598    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.179306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179188    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:35.179423 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179315    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home" (OuterVolumeSpecName: "home") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:35.179423 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179372    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:35.179684 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179636    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm" (OuterVolumeSpecName: "dshm") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:35.179852 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179831    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf" (OuterVolumeSpecName: "kube-api-access-4dsbf") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "kube-api-access-4dsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:35.179900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.179834    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c83db231-7174-4417-8c78-02c8bb08eca1" (UID: "c83db231-7174-4417-8c78-02c8bb08eca1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:35.180390 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.180375    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.181185 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.181169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.187855 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.187835    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksb9\" (UniqueName: \"kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9\") pod \"late-v1-kserve-56cbc8456c-gcmkw\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.278621 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278589    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.278621 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278615    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.278621 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278625    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.278855 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278634    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83db231-7174-4417-8c78-02c8bb08eca1-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.278855 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278644    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dsbf\" (UniqueName: \"kubernetes.io/projected/c83db231-7174-4417-8c78-02c8bb08eca1-kube-api-access-4dsbf\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.278855 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.278651    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83db231-7174-4417-8c78-02c8bb08eca1-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:22:35.316187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.316118    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:35.452575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.452549    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:22:35.452977 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:22:35.452958    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816d00c5_2c09_42cd_ae48_849c39476878.slice/crio-9f34350171c177cee5c10eb7a4299a3a84c9edea97e536d47c2b2b74ddb74a4c WatchSource:0}: Error finding container 9f34350171c177cee5c10eb7a4299a3a84c9edea97e536d47c2b2b74ddb74a4c: Status 404 returned error can't find the container with id 9f34350171c177cee5c10eb7a4299a3a84c9edea97e536d47c2b2b74ddb74a4c
Jul 31 15:22:35.947276 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.947214    2557 generic.go:356] "Generic (PLEG): container finished" podID="c83db231-7174-4417-8c78-02c8bb08eca1" containerID="77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183" exitCode=0
Jul 31 15:22:35.947729 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.947286    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"
Jul 31 15:22:35.947729 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.947297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" event={"ID":"c83db231-7174-4417-8c78-02c8bb08eca1","Type":"ContainerDied","Data":"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"}
Jul 31 15:22:35.947729 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.947352    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj" event={"ID":"c83db231-7174-4417-8c78-02c8bb08eca1","Type":"ContainerDied","Data":"992d931940bb1c69987f6a027fb1bbecb9340b5d6b22e088a2d9876427ae1c9a"}
Jul 31 15:22:35.947729 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.947374    2557 scope.go:117] "RemoveContainer" containerID="77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"
Jul 31 15:22:35.949037 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.949016    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" event={"ID":"816d00c5-2c09-42cd-ae48-849c39476878","Type":"ContainerStarted","Data":"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"}
Jul 31 15:22:35.949153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.949060    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" event={"ID":"816d00c5-2c09-42cd-ae48-849c39476878","Type":"ContainerStarted","Data":"9f34350171c177cee5c10eb7a4299a3a84c9edea97e536d47c2b2b74ddb74a4c"}
Jul 31 15:22:35.956367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.956351    2557 scope.go:117] "RemoveContainer" containerID="77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"
Jul 31 15:22:35.956615 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:22:35.956596    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183\": container with ID starting with 77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183 not found: ID does not exist" containerID="77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"
Jul 31 15:22:35.956688 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.956623    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183"} err="failed to get container status \"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183\": rpc error: code = NotFound desc = could not find container \"77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183\": container with ID starting with 77266a5345560d96a9a38c08be9b323c8f25241af6db0bea497d3af0a1752183 not found: ID does not exist"
Jul 31 15:22:35.970387 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.970335    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" podStartSLOduration=1.970321356 podStartE2EDuration="1.970321356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:22:35.969882855 +0000 UTC m=+2078.128134344" watchObservedRunningTime="2026-07-31 15:22:35.970321356 +0000 UTC m=+2078.128572845"
Jul 31 15:22:35.985526 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.985489    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:22:35.990378 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:35.990346    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-7pwvj"]
Jul 31 15:22:36.247933 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:36.247856    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372bae9f-9fa4-4cef-9588-37d72f07fa50" path="/var/lib/kubelet/pods/372bae9f-9fa4-4cef-9588-37d72f07fa50/volumes"
Jul 31 15:22:36.248226 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:36.248213    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c83db231-7174-4417-8c78-02c8bb08eca1" path="/var/lib/kubelet/pods/c83db231-7174-4417-8c78-02c8bb08eca1/volumes"
Jul 31 15:22:36.316500 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:36.316467    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:45.316314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:45.316278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:45.328378 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:45.328350    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:46.006969 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:46.006940    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:22:58.324753 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:58.324717    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:22:58.331289 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:22:58.331262    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:23:09.109229 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.109194    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:23:09.109699 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.109599    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c83db231-7174-4417-8c78-02c8bb08eca1" containerName="main"
Jul 31 15:23:09.109699 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.109612    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c83db231-7174-4417-8c78-02c8bb08eca1" containerName="main"
Jul 31 15:23:09.109699 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.109676    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c83db231-7174-4417-8c78-02c8bb08eca1" containerName="main"
Jul 31 15:23:09.112247 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.112217    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.116162 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.116142    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-zrs6n\""
Jul 31 15:23:09.116162 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.116156    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 31 15:23:09.116383 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.116156    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 31 15:23:09.116383 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.116156    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 31 15:23:09.123606 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.123586    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:23:09.192523 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192484    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.192728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192538    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlnl\" (UniqueName: \"kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.192728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192559    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.192728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192628    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.192728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.192728 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.192724    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.293746 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.293708    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.293746 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.293764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294036 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.293889    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlnl\" (UniqueName: \"kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294036 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.293949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294036 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.293992    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294036 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.294010    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294278 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.294172    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294420 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.294396    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.294475 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.294430    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.296072 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.296051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.296345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.296329    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.302678 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.302652    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlnl\" (UniqueName: \"kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.422670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.422628    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:09.555810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:09.555781    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:23:09.558109 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:23:09.558085    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a1a794_c95d_4dfa_a9b3_a3fb7890af7f.slice/crio-ca866cd6f755cf2d15b8dd0bc603718206355a212b1e2f05af5aeec103cd021b WatchSource:0}: Error finding container ca866cd6f755cf2d15b8dd0bc603718206355a212b1e2f05af5aeec103cd021b: Status 404 returned error can't find the container with id ca866cd6f755cf2d15b8dd0bc603718206355a212b1e2f05af5aeec103cd021b
Jul 31 15:23:10.111349 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:10.111310    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" event={"ID":"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f","Type":"ContainerStarted","Data":"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"}
Jul 31 15:23:10.111810 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:10.111372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" event={"ID":"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f","Type":"ContainerStarted","Data":"ca866cd6f755cf2d15b8dd0bc603718206355a212b1e2f05af5aeec103cd021b"}
Jul 31 15:23:10.133865 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:10.133814    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" podStartSLOduration=1.133796062 podStartE2EDuration="1.133796062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:23:10.132421175 +0000 UTC m=+2112.290672664" watchObservedRunningTime="2026-07-31 15:23:10.133796062 +0000 UTC m=+2112.292047551"
Jul 31 15:23:10.422938 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:10.422909    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:19.423592 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:19.423554    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:19.435793 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:19.435769    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:20.170670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:20.170635    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:23:33.606433 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.606396    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:23:33.609779 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.609761    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.612559 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.612536    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Jul 31 15:23:33.627922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.627885    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:23:33.718514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718473    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.718514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718516    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.718787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.718787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718652    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.718787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.718787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.718764    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.819873 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.819839    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.819900    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.819928    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.819949    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.819995    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.820022    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820371 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.820328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820417 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.820374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.820451 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.820432    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.822183 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.822163    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.822364 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.822342    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.828487 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.828467    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn\") pod \"late-v2-kserve-86766546f-x462q\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:33.924491 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:33.924462    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:34.058900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:34.058873    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:23:34.061753 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:23:34.061726    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a6ab87_8e9a_4716_be78_ff3333197857.slice/crio-0a3f35e38edcfcc7db03b33db2045ec604ad91a55350fda5422fe8e500285b61 WatchSource:0}: Error finding container 0a3f35e38edcfcc7db03b33db2045ec604ad91a55350fda5422fe8e500285b61: Status 404 returned error can't find the container with id 0a3f35e38edcfcc7db03b33db2045ec604ad91a55350fda5422fe8e500285b61
Jul 31 15:23:34.226821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:34.226731    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" event={"ID":"b4a6ab87-8e9a-4716-be78-ff3333197857","Type":"ContainerStarted","Data":"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"}
Jul 31 15:23:34.226821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:34.226791    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" event={"ID":"b4a6ab87-8e9a-4716-be78-ff3333197857","Type":"ContainerStarted","Data":"0a3f35e38edcfcc7db03b33db2045ec604ad91a55350fda5422fe8e500285b61"}
Jul 31 15:23:34.247178 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:34.247129    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" podStartSLOduration=1.24711425 podStartE2EDuration="1.24711425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:23:34.245655402 +0000 UTC m=+2136.403906920" watchObservedRunningTime="2026-07-31 15:23:34.24711425 +0000 UTC m=+2136.405365738"
Jul 31 15:23:34.925556 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:34.925514    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:43.924744 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:43.924709    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:43.936609 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:43.936579    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:44.281284 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:44.281173    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:23:59.827854 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:59.827762    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:23:59.838968 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:23:59.838919    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:23:59.838968 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:23:59.838968 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:23:59.838968 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36" cmd=["/bin/sleep","15"]
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:59.838994    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" message="" err=<
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:23:59.839011    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main" containerID="cri-o://f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"
Jul 31 15:23:59.839314 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:23:59.839057    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main" containerID="cri-o://f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36" gracePeriod=60
Jul 31 15:24:00.190501 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.190476    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:24:00.257868 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.257834    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.257909    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.257932    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.257957    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.257996    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdlnl\" (UniqueName: \"kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258047    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm\") pod \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\" (UID: \"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f\") "
Jul 31 15:24:00.258341 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258205    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:00.258341 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258202    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home" (OuterVolumeSpecName: "home") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:00.258341 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258259    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:00.258460 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258363    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.258460 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258376    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.258460 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.258389    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.260319 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.260294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:00.260406 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.260294    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm" (OuterVolumeSpecName: "dshm") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:00.260406 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.260386    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl" (OuterVolumeSpecName: "kube-api-access-vdlnl") pod "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" (UID: "e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f"). InnerVolumeSpecName "kube-api-access-vdlnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:00.345520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.345487    2557 generic.go:356] "Generic (PLEG): container finished" podID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerID="f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36" exitCode=0
Jul 31 15:24:00.345698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.345538    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"
Jul 31 15:24:00.345698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.345555    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" event={"ID":"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f","Type":"ContainerDied","Data":"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"}
Jul 31 15:24:00.345698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.345599    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" event={"ID":"e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f","Type":"ContainerDied","Data":"ca866cd6f755cf2d15b8dd0bc603718206355a212b1e2f05af5aeec103cd021b"}
Jul 31 15:24:00.345698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.345620    2557 scope.go:117] "RemoveContainer" containerID="f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"
Jul 31 15:24:00.355125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.355107    2557 scope.go:117] "RemoveContainer" containerID="f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"
Jul 31 15:24:00.355422 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:24:00.355404    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36\": container with ID starting with f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36 not found: ID does not exist" containerID="f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"
Jul 31 15:24:00.355498 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.355429    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36"} err="failed to get container status \"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36\": rpc error: code = NotFound desc = could not find container \"f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36\": container with ID starting with f14fbc7bbe9b118a5bb7d8aeaa18472d3ce90335d6a39ddea0a2add36a12ac36 not found: ID does not exist"
Jul 31 15:24:00.359126 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.359082    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.359126 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.359104    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlnl\" (UniqueName: \"kubernetes.io/projected/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-kube-api-access-vdlnl\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.359126 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.359115    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:00.367099 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.367072    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:24:00.371832 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:00.371808    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb"]
Jul 31 15:24:01.160716 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:01.160660    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-s6qcb" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Jul 31 15:24:02.247958 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:02.247924    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" path="/var/lib/kubelet/pods/e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f/volumes"
Jul 31 15:24:32.848819 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:32.848781    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:24:32.854413 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:32.854380    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:24:47.868026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:47.867983    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" podUID="816d00c5-2c09-42cd-ae48-849c39476878" containerName="main" containerID="cri-o://c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6" gracePeriod=45
Jul 31 15:24:47.872657 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:47.872616    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" podUID="b4a6ab87-8e9a-4716-be78-ff3333197857" containerName="main" containerID="cri-o://cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0" gracePeriod=45
Jul 31 15:24:47.997011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:47.996967    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" podUID="816d00c5-2c09-42cd-ae48-849c39476878" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Jul 31 15:24:48.131350 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.131324    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:24:48.134094 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.134075    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:24:48.189843 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189816    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189860    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189912    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189927    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189953    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189973    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.189996    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190049    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190076    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pksb9\" (UniqueName: \"kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190102    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190143    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir\") pod \"816d00c5-2c09-42cd-ae48-849c39476878\" (UID: \"816d00c5-2c09-42cd-ae48-849c39476878\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190192    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home\") pod \"b4a6ab87-8e9a-4716-be78-ff3333197857\" (UID: \"b4a6ab87-8e9a-4716-be78-ff3333197857\") "
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190214    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache" (OuterVolumeSpecName: "model-cache") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.190346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190229    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home" (OuterVolumeSpecName: "home") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.190727 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190501    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.190727 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190521    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.190835 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190740    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home" (OuterVolumeSpecName: "home") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.190835 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190741    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.190938 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.190818    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.191139 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.191117    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.192726 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.192696    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm" (OuterVolumeSpecName: "dshm") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.192834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.192786    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn" (OuterVolumeSpecName: "kube-api-access-xvqhn") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "kube-api-access-xvqhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:48.192905 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.192883    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9" (OuterVolumeSpecName: "kube-api-access-pksb9") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "kube-api-access-pksb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:24:48.193027 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.193006    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:48.193299 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.193280    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm" (OuterVolumeSpecName: "dshm") pod "816d00c5-2c09-42cd-ae48-849c39476878" (UID: "816d00c5-2c09-42cd-ae48-849c39476878"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:24:48.193350 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.193306    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4a6ab87-8e9a-4716-be78-ff3333197857" (UID: "b4a6ab87-8e9a-4716-be78-ff3333197857"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:24:48.291948 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291912    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.291948 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291939    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.291948 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291947    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a6ab87-8e9a-4716-be78-ff3333197857-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.291948 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291957    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816d00c5-2c09-42cd-ae48-849c39476878-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291985    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.291994    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvqhn\" (UniqueName: \"kubernetes.io/projected/b4a6ab87-8e9a-4716-be78-ff3333197857-kube-api-access-xvqhn\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.292002    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.292010    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksb9\" (UniqueName: \"kubernetes.io/projected/816d00c5-2c09-42cd-ae48-849c39476878-kube-api-access-pksb9\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.292018    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a6ab87-8e9a-4716-be78-ff3333197857-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.292213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.292027    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816d00c5-2c09-42cd-ae48-849c39476878-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:24:48.575999 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.575966    2557 generic.go:356] "Generic (PLEG): container finished" podID="b4a6ab87-8e9a-4716-be78-ff3333197857" containerID="cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0" exitCode=0
Jul 31 15:24:48.576187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.576025    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"
Jul 31 15:24:48.576187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.576057    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" event={"ID":"b4a6ab87-8e9a-4716-be78-ff3333197857","Type":"ContainerDied","Data":"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"}
Jul 31 15:24:48.576187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.576115    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q" event={"ID":"b4a6ab87-8e9a-4716-be78-ff3333197857","Type":"ContainerDied","Data":"0a3f35e38edcfcc7db03b33db2045ec604ad91a55350fda5422fe8e500285b61"}
Jul 31 15:24:48.576187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.576144    2557 scope.go:117] "RemoveContainer" containerID="cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"
Jul 31 15:24:48.577644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.577613    2557 generic.go:356] "Generic (PLEG): container finished" podID="816d00c5-2c09-42cd-ae48-849c39476878" containerID="c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6" exitCode=0
Jul 31 15:24:48.577771 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.577644    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" event={"ID":"816d00c5-2c09-42cd-ae48-849c39476878","Type":"ContainerDied","Data":"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"}
Jul 31 15:24:48.577771 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.577690    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw" event={"ID":"816d00c5-2c09-42cd-ae48-849c39476878","Type":"ContainerDied","Data":"9f34350171c177cee5c10eb7a4299a3a84c9edea97e536d47c2b2b74ddb74a4c"}
Jul 31 15:24:48.577771 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.577737    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"
Jul 31 15:24:48.587897 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.587875    2557 scope.go:117] "RemoveContainer" containerID="cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"
Jul 31 15:24:48.588380 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:24:48.588357    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0\": container with ID starting with cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0 not found: ID does not exist" containerID="cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"
Jul 31 15:24:48.588483 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.588386    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0"} err="failed to get container status \"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0\": rpc error: code = NotFound desc = could not find container \"cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0\": container with ID starting with cc09ba333fd95d44a25a90b25ba0d7f4fd5892c95276817d7f9ea89dc7e68db0 not found: ID does not exist"
Jul 31 15:24:48.588483 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.588402    2557 scope.go:117] "RemoveContainer" containerID="c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"
Jul 31 15:24:48.596145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.596127    2557 scope.go:117] "RemoveContainer" containerID="c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"
Jul 31 15:24:48.596431 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:24:48.596408    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6\": container with ID starting with c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6 not found: ID does not exist" containerID="c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"
Jul 31 15:24:48.596511 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.596436    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6"} err="failed to get container status \"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6\": rpc error: code = NotFound desc = could not find container \"c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6\": container with ID starting with c40a311946fdc9744556986765fecaa4b517429dd53f128d360f732c6182f9b6 not found: ID does not exist"
Jul 31 15:24:48.596712 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.596693    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:24:48.600469 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.600449    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-x462q"]
Jul 31 15:24:48.610409 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.610389    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:24:48.616954 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:48.616934    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-gcmkw"]
Jul 31 15:24:49.197256 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197216    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:24:49.197692 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197659    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="816d00c5-2c09-42cd-ae48-849c39476878" containerName="main"
Jul 31 15:24:49.197692 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197676    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="816d00c5-2c09-42cd-ae48-849c39476878" containerName="main"
Jul 31 15:24:49.197692 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197692    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197702    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197711    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a6ab87-8e9a-4716-be78-ff3333197857" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197719    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a6ab87-8e9a-4716-be78-ff3333197857" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197787    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="816d00c5-2c09-42cd-ae48-849c39476878" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197799    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a6ab87-8e9a-4716-be78-ff3333197857" containerName="main"
Jul 31 15:24:49.197875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.197805    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a1a794-c95d-4dfa-a9b3-a3fb7890af7f" containerName="main"
Jul 31 15:24:49.202465 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.202444    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.205796 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.205773    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Jul 31 15:24:49.205927 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.205797    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Jul 31 15:24:49.207045 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.207025    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Jul 31 15:24:49.207146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.207053    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-48cmm\""
Jul 31 15:24:49.212838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.212815    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:24:49.299902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.299860    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.300077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.299906    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhs48\" (UniqueName: \"kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.300077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.299976    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.300077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.300045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.300181 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.300107    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.300181 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.300137    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401480 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401442    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401480 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401723 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401511    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhs48\" (UniqueName: \"kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401723 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401545    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401723 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.401868 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.401726    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.402021 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.402002    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.402084 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.402005    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.402084 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.402051    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.403784 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.403760    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.404041 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.404024    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.413099 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.413077    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhs48\" (UniqueName: \"kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48\") pod \"delw-v2-kserve-5b84cc7ddf-sntwd\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.513872 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.513773    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:49.666005 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:49.665977    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:24:49.668861 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:24:49.668834    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd824894c_a1fd_4af0_911d_3c11bd4f79f7.slice/crio-f02b9139318747ea58e67a14f4e1513b399716dc404ddbaf327f288be8e8d6a4 WatchSource:0}: Error finding container f02b9139318747ea58e67a14f4e1513b399716dc404ddbaf327f288be8e8d6a4: Status 404 returned error can't find the container with id f02b9139318747ea58e67a14f4e1513b399716dc404ddbaf327f288be8e8d6a4
Jul 31 15:24:50.248105 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:50.248071    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="816d00c5-2c09-42cd-ae48-849c39476878" path="/var/lib/kubelet/pods/816d00c5-2c09-42cd-ae48-849c39476878/volumes"
Jul 31 15:24:50.248529 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:50.248514    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a6ab87-8e9a-4716-be78-ff3333197857" path="/var/lib/kubelet/pods/b4a6ab87-8e9a-4716-be78-ff3333197857/volumes"
Jul 31 15:24:50.590869 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:50.590764    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" event={"ID":"d824894c-a1fd-4af0-911d-3c11bd4f79f7","Type":"ContainerStarted","Data":"83d1d57f678cf97870e4c53b85f1ee5afd9ac7f1bc4a7e81c231a0ea4d5cd39c"}
Jul 31 15:24:50.590869 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:50.590838    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" event={"ID":"d824894c-a1fd-4af0-911d-3c11bd4f79f7","Type":"ContainerStarted","Data":"f02b9139318747ea58e67a14f4e1513b399716dc404ddbaf327f288be8e8d6a4"}
Jul 31 15:24:50.610209 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:50.610161    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" podStartSLOduration=1.610144231 podStartE2EDuration="1.610144231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:24:50.608849945 +0000 UTC m=+2212.767101433" watchObservedRunningTime="2026-07-31 15:24:50.610144231 +0000 UTC m=+2212.768395722"
Jul 31 15:24:51.514338 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:51.514305    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:59.514842 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:59.514799    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:59.526923 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:59.526895    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:24:59.642403 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:24:59.642372    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:25:51.703648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:25:51.703616    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:26:06.722213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:06.722169    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" podUID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" containerName="main" containerID="cri-o://83d1d57f678cf97870e4c53b85f1ee5afd9ac7f1bc4a7e81c231a0ea4d5cd39c" gracePeriod=45
Jul 31 15:26:06.944991 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:06.944957    2557 generic.go:356] "Generic (PLEG): container finished" podID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" containerID="83d1d57f678cf97870e4c53b85f1ee5afd9ac7f1bc4a7e81c231a0ea4d5cd39c" exitCode=0
Jul 31 15:26:06.945165 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:06.945032    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" event={"ID":"d824894c-a1fd-4af0-911d-3c11bd4f79f7","Type":"ContainerDied","Data":"83d1d57f678cf97870e4c53b85f1ee5afd9ac7f1bc4a7e81c231a0ea4d5cd39c"}
Jul 31 15:26:06.969126 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:06.969095    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:26:07.092532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092446    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092498    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhs48\" (UniqueName: \"kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092532 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092524    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092548    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092586    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092821 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092645    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir\") pod \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\" (UID: \"d824894c-a1fd-4af0-911d-3c11bd4f79f7\") "
Jul 31 15:26:07.092967 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092861    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home" (OuterVolumeSpecName: "home") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:07.092967 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092893    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:07.092967 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.092925    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:07.094762 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.094734    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm" (OuterVolumeSpecName: "dshm") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:26:07.094902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.094803    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48" (OuterVolumeSpecName: "kube-api-access-fhs48") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "kube-api-access-fhs48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:26:07.094902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.094825    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d824894c-a1fd-4af0-911d-3c11bd4f79f7" (UID: "d824894c-a1fd-4af0-911d-3c11bd4f79f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:26:07.193703 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193662    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.193703 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193698    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.193703 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193711    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhs48\" (UniqueName: \"kubernetes.io/projected/d824894c-a1fd-4af0-911d-3c11bd4f79f7-kube-api-access-fhs48\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.193983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193725    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d824894c-a1fd-4af0-911d-3c11bd4f79f7-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.193983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193739    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.193983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.193754    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d824894c-a1fd-4af0-911d-3c11bd4f79f7-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:26:07.951392 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.951358    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd" event={"ID":"d824894c-a1fd-4af0-911d-3c11bd4f79f7","Type":"ContainerDied","Data":"f02b9139318747ea58e67a14f4e1513b399716dc404ddbaf327f288be8e8d6a4"}
Jul 31 15:26:07.951857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.951414    2557 scope.go:117] "RemoveContainer" containerID="83d1d57f678cf97870e4c53b85f1ee5afd9ac7f1bc4a7e81c231a0ea4d5cd39c"
Jul 31 15:26:07.951857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.951370    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"
Jul 31 15:26:07.975471 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.975444    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:26:07.980543 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:07.980514    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-sntwd"]
Jul 31 15:26:08.248731 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:08.248649    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" path="/var/lib/kubelet/pods/d824894c-a1fd-4af0-911d-3c11bd4f79f7/volumes"
Jul 31 15:26:32.911454 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.911414    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:26:32.911918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.911749    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" containerName="main"
Jul 31 15:26:32.911918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.911760    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" containerName="main"
Jul 31 15:26:32.911918 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.911826    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d824894c-a1fd-4af0-911d-3c11bd4f79f7" containerName="main"
Jul 31 15:26:32.916041 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.916024    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:32.918719 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.918699    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Jul 31 15:26:32.919684 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.919664    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Jul 31 15:26:32.919792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.919734    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Jul 31 15:26:32.919792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.919744    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-wdmnj\""
Jul 31 15:26:32.924953 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:32.924932    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:26:33.030705 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030670    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.030926 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030710    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9kpv\" (UniqueName: \"kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.030926 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030735    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.030926 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030841    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.030926 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030887    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.030926 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.030907    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131564 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131522    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131575    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131600    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131676    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131715    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9kpv\" (UniqueName: \"kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.131761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.131746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.132037 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.132014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.132090 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.132029    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.132274 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.132254    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.133949 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.133922    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.134290 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.134273    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.139732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.139712    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9kpv\" (UniqueName: \"kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv\") pod \"roll-v1-kserve-69cf8bcfc-g5ml6\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.226369 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.226323    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:33.367836 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:33.367804    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:26:33.370822 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:26:33.370790    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ecdad7_f5c8_4dcc_9a0d_b8960fd83819.slice/crio-3be203454b02c75257274efcac08ae9abe04ccec145ba6f306692fa2904bd3f3 WatchSource:0}: Error finding container 3be203454b02c75257274efcac08ae9abe04ccec145ba6f306692fa2904bd3f3: Status 404 returned error can't find the container with id 3be203454b02c75257274efcac08ae9abe04ccec145ba6f306692fa2904bd3f3
Jul 31 15:26:34.075533 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:34.075499    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" event={"ID":"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819","Type":"ContainerStarted","Data":"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"}
Jul 31 15:26:34.075934 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:34.075550    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" event={"ID":"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819","Type":"ContainerStarted","Data":"3be203454b02c75257274efcac08ae9abe04ccec145ba6f306692fa2904bd3f3"}
Jul 31 15:26:34.093985 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:34.093928    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" podStartSLOduration=2.093906287 podStartE2EDuration="2.093906287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:26:34.092936143 +0000 UTC m=+2316.251187657" watchObservedRunningTime="2026-07-31 15:26:34.093906287 +0000 UTC m=+2316.252157777"
Jul 31 15:26:34.226592 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:34.226558    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:43.227320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:43.227278    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:43.239429 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:43.239404    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:44.131544 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:44.131510    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:26:45.606397 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.606353    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:26:45.609737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.609718    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.612145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.612127    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Jul 31 15:26:45.622253 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.619066    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:26:45.745170 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745132    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.745170 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745173    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqz9\" (UniqueName: \"kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.745443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745198    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.745443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745335    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.745443 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745370    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.745561 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.745443    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846184 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846147    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846184 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846187    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846472 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846221    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846472 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846269    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846472 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846295    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqz9\" (UniqueName: \"kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846472 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846318    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846679 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846637    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846735 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846683    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.846783 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.846766    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.848602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.848577    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.848805 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.848789    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.855091 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.855071    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqz9\" (UniqueName: \"kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9\") pod \"roll-v2-kserve-d4dbf9d67-7z82c\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:45.925795 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:45.925763    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:46.061150 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:46.061121    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:26:46.063757 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:26:46.063727    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a449bb_d351_458a_ba9f_63accb5ae170.slice/crio-51fa9006b57e55aa96c16ba397ff6f03341cda8c7550620fed133412364cfb28 WatchSource:0}: Error finding container 51fa9006b57e55aa96c16ba397ff6f03341cda8c7550620fed133412364cfb28: Status 404 returned error can't find the container with id 51fa9006b57e55aa96c16ba397ff6f03341cda8c7550620fed133412364cfb28
Jul 31 15:26:46.133044 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:46.133008    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" event={"ID":"b2a449bb-d351-458a-ba9f-63accb5ae170","Type":"ContainerStarted","Data":"51fa9006b57e55aa96c16ba397ff6f03341cda8c7550620fed133412364cfb28"}
Jul 31 15:26:46.149990 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:46.149933    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" podStartSLOduration=1.149915618 podStartE2EDuration="1.149915618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:26:46.149400506 +0000 UTC m=+2328.307651994" watchObservedRunningTime="2026-07-31 15:26:46.149915618 +0000 UTC m=+2328.308167106"
Jul 31 15:26:46.926814 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:46.926778    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:47.138655 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:47.138622    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" event={"ID":"b2a449bb-d351-458a-ba9f-63accb5ae170","Type":"ContainerStarted","Data":"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"}
Jul 31 15:26:55.926787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:55.926687    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:55.938848 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:55.938821    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:26:56.189273 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:26:56.189172    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:27:58.353726 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:27:58.353688    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:27:58.364943 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:27:58.364913    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:28:16.626265 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:16.626214    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:28:16.630280 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:16.630253    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:28:31.645344 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:31.645219    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" podUID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" containerName="main" containerID="cri-o://af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029" gracePeriod=45
Jul 31 15:28:31.648563 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:31.648531    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" podUID="b2a449bb-d351-458a-ba9f-63accb5ae170" containerName="main" containerID="cri-o://c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81" gracePeriod=45
Jul 31 15:28:31.927180 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:31.927155    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:28:31.929788 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:31.929768    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:28:32.021682 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021642    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021693    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021715    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021784    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021809    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9kpv\" (UniqueName: \"kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021841    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.021889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021873    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021911    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir\") pod \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\" (UID: \"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021915    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home" (OuterVolumeSpecName: "home") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021935    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021967    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.021948    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache" (OuterVolumeSpecName: "model-cache") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022034    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022036    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home" (OuterVolumeSpecName: "home") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022071    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggqz9\" (UniqueName: \"kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9\") pod \"b2a449bb-d351-458a-ba9f-63accb5ae170\" (UID: \"b2a449bb-d351-458a-ba9f-63accb5ae170\") "
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022088    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022220 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022155    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022282    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022419    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022439    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022454    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022467    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022480    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.022718 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.022492    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.024363 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024335    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9" (OuterVolumeSpecName: "kube-api-access-ggqz9") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "kube-api-access-ggqz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:28:32.024512 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024484    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv" (OuterVolumeSpecName: "kube-api-access-x9kpv") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "kube-api-access-x9kpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:28:32.024572 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024505    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:28:32.024572 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024527    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm" (OuterVolumeSpecName: "dshm") pod "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" (UID: "07ecdad7-f5c8-4dcc-9a0d-b8960fd83819"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.024645 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024572    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm" (OuterVolumeSpecName: "dshm") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:32.024727 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.024710    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2a449bb-d351-458a-ba9f-63accb5ae170" (UID: "b2a449bb-d351-458a-ba9f-63accb5ae170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:28:32.123672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123635    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2a449bb-d351-458a-ba9f-63accb5ae170-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.123672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123666    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2a449bb-d351-458a-ba9f-63accb5ae170-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.123672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123677    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqz9\" (UniqueName: \"kubernetes.io/projected/b2a449bb-d351-458a-ba9f-63accb5ae170-kube-api-access-ggqz9\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.123672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123686    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.123945 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123694    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9kpv\" (UniqueName: \"kubernetes.io/projected/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-kube-api-access-x9kpv\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.123945 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.123703    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:28:32.628495 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.628409    2557 generic.go:356] "Generic (PLEG): container finished" podID="b2a449bb-d351-458a-ba9f-63accb5ae170" containerID="c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81" exitCode=0
Jul 31 15:28:32.628495 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.628456    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"
Jul 31 15:28:32.628722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.628490    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" event={"ID":"b2a449bb-d351-458a-ba9f-63accb5ae170","Type":"ContainerDied","Data":"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"}
Jul 31 15:28:32.628722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.628537    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c" event={"ID":"b2a449bb-d351-458a-ba9f-63accb5ae170","Type":"ContainerDied","Data":"51fa9006b57e55aa96c16ba397ff6f03341cda8c7550620fed133412364cfb28"}
Jul 31 15:28:32.628722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.628564    2557 scope.go:117] "RemoveContainer" containerID="c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"
Jul 31 15:28:32.629823 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.629800    2557 generic.go:356] "Generic (PLEG): container finished" podID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" containerID="af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029" exitCode=0
Jul 31 15:28:32.629930 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.629828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" event={"ID":"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819","Type":"ContainerDied","Data":"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"}
Jul 31 15:28:32.629930 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.629851    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"
Jul 31 15:28:32.629930 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.629874    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6" event={"ID":"07ecdad7-f5c8-4dcc-9a0d-b8960fd83819","Type":"ContainerDied","Data":"3be203454b02c75257274efcac08ae9abe04ccec145ba6f306692fa2904bd3f3"}
Jul 31 15:28:32.636928 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.636911    2557 scope.go:117] "RemoveContainer" containerID="c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"
Jul 31 15:28:32.637157 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:28:32.637141    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81\": container with ID starting with c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81 not found: ID does not exist" containerID="c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"
Jul 31 15:28:32.637203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.637168    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81"} err="failed to get container status \"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81\": rpc error: code = NotFound desc = could not find container \"c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81\": container with ID starting with c30bf71fed653444f230cd850de6e2e8e1440096692af4d64d4fe0b9b7fd2b81 not found: ID does not exist"
Jul 31 15:28:32.637203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.637185    2557 scope.go:117] "RemoveContainer" containerID="af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"
Jul 31 15:28:32.644860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.644839    2557 scope.go:117] "RemoveContainer" containerID="af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"
Jul 31 15:28:32.645135 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:28:32.645116    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029\": container with ID starting with af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029 not found: ID does not exist" containerID="af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"
Jul 31 15:28:32.645184 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.645142    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029"} err="failed to get container status \"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029\": rpc error: code = NotFound desc = could not find container \"af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029\": container with ID starting with af7563194142ef5b7e5ae4aa9419895d7a1566ddf6e186e60d362705fba7d029 not found: ID does not exist"
Jul 31 15:28:32.646102 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.646083    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:28:32.649070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.649050    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-g5ml6"]
Jul 31 15:28:32.659672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.659653    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:28:32.663248 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:32.663216    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-7z82c"]
Jul 31 15:28:34.248269 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:34.248220    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" path="/var/lib/kubelet/pods/07ecdad7-f5c8-4dcc-9a0d-b8960fd83819/volumes"
Jul 31 15:28:34.248660 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:34.248639    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a449bb-d351-458a-ba9f-63accb5ae170" path="/var/lib/kubelet/pods/b2a449bb-d351-458a-ba9f-63accb5ae170/volumes"
Jul 31 15:28:40.115775 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.115741    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116078    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a449bb-d351-458a-ba9f-63accb5ae170" containerName="main"
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116089    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a449bb-d351-458a-ba9f-63accb5ae170" containerName="main"
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116107    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" containerName="main"
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116112    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" containerName="main"
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116167    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ecdad7-f5c8-4dcc-9a0d-b8960fd83819" containerName="main"
Jul 31 15:28:40.116201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.116173    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a449bb-d351-458a-ba9f-63accb5ae170" containerName="main"
Jul 31 15:28:40.119184 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.119167    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.123687 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.123658    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Jul 31 15:28:40.124902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.124652    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Jul 31 15:28:40.124902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.124872    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-n7v8j\""
Jul 31 15:28:40.125073 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.125004    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Jul 31 15:28:40.130756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.130735    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:28:40.189744 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189706    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.189938 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189762    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.189938 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btzx\" (UniqueName: \"kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.189938 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189886    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.190062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189938    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.190062 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.189965    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290458 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290419    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290476    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290590    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btzx\" (UniqueName: \"kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290625    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290785 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290673    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290785 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290697    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.290874 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.291013 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.290991    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.291065 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.291028    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.292830 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.292804    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.292994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.292976    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.301889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.301866    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btzx\" (UniqueName: \"kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx\") pod \"owner-v2-kserve-5cf8959f95-j7qbt\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.429836 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.429801    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:40.565011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.564984    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:28:40.567120 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:28:40.567089    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d19e2ca_6c4e_4baf_b9a6_d7042f58b1d6.slice/crio-e7f6b9c6d2b22ba85199c9e200e284ad24d673fa0345ffbbacc0f307a3954265 WatchSource:0}: Error finding container e7f6b9c6d2b22ba85199c9e200e284ad24d673fa0345ffbbacc0f307a3954265: Status 404 returned error can't find the container with id e7f6b9c6d2b22ba85199c9e200e284ad24d673fa0345ffbbacc0f307a3954265
Jul 31 15:28:40.670057 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.670004    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" event={"ID":"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6","Type":"ContainerStarted","Data":"f86ef7e435f886f3458678591203014d6975a336f4ed70cabc0650c733c457d3"}
Jul 31 15:28:40.670280 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.670078    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" event={"ID":"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6","Type":"ContainerStarted","Data":"e7f6b9c6d2b22ba85199c9e200e284ad24d673fa0345ffbbacc0f307a3954265"}
Jul 31 15:28:40.695451 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:40.695358    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" podStartSLOduration=0.695341662 podStartE2EDuration="695.341662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:28:40.694268807 +0000 UTC m=+2442.852520288" watchObservedRunningTime="2026-07-31 15:28:40.695341662 +0000 UTC m=+2442.853593183"
Jul 31 15:28:41.430945 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:41.430900    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:50.430791 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:50.430745    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:50.442982 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:50.442958    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:28:50.726667 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:28:50.726591    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:29:48.792182 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:29:48.792101    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:30:03.811710 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:03.811646    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" podUID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" containerName="main" containerID="cri-o://f86ef7e435f886f3458678591203014d6975a336f4ed70cabc0650c733c457d3" gracePeriod=45
Jul 31 15:30:04.046203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.046168    2557 generic.go:356] "Generic (PLEG): container finished" podID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" containerID="f86ef7e435f886f3458678591203014d6975a336f4ed70cabc0650c733c457d3" exitCode=0
Jul 31 15:30:04.046352 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.046247    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" event={"ID":"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6","Type":"ContainerDied","Data":"f86ef7e435f886f3458678591203014d6975a336f4ed70cabc0650c733c457d3"}
Jul 31 15:30:04.061892 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.061834    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:30:04.130445 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130414    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130635 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130477    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8btzx\" (UniqueName: \"kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130635 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130562    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130635 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130584    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130635 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130605    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130835 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130642    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir\") pod \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\" (UID: \"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6\") "
Jul 31 15:30:04.130903 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130866    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home" (OuterVolumeSpecName: "home") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:04.130957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130877    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:04.130998 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.130958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:04.132618 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.132592    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm" (OuterVolumeSpecName: "dshm") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:30:04.132721 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.132627    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:30:04.132721 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.132640    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx" (OuterVolumeSpecName: "kube-api-access-8btzx") pod "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" (UID: "2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6"). InnerVolumeSpecName "kube-api-access-8btzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:30:04.232287 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232211    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:04.232287 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232280    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:04.232287 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232290    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btzx\" (UniqueName: \"kubernetes.io/projected/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-kube-api-access-8btzx\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:04.232287 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232301    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:04.232565 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232310    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:04.232565 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:04.232317    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:30:05.051451 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:05.051417    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"
Jul 31 15:30:05.051992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:05.051450    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt" event={"ID":"2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6","Type":"ContainerDied","Data":"e7f6b9c6d2b22ba85199c9e200e284ad24d673fa0345ffbbacc0f307a3954265"}
Jul 31 15:30:05.051992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:05.051506    2557 scope.go:117] "RemoveContainer" containerID="f86ef7e435f886f3458678591203014d6975a336f4ed70cabc0650c733c457d3"
Jul 31 15:30:05.069650 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:05.069625    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:30:05.073340 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:05.073318    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-j7qbt"]
Jul 31 15:30:06.247462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:06.247429    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" path="/var/lib/kubelet/pods/2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6/volumes"
Jul 31 15:30:39.006790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.006754    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:30:39.007265 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.007154    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" containerName="main"
Jul 31 15:30:39.007265 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.007167    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" containerName="main"
Jul 31 15:30:39.007265 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.007263    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d19e2ca-6c4e-4baf-b9a6-d7042f58b1d6" containerName="main"
Jul 31 15:30:39.010499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.010477    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.013076 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.013050    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Jul 31 15:30:39.013877 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.013853    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 31 15:30:39.013989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.013876    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Jul 31 15:30:39.013989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.013899    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-qs9tn\""
Jul 31 15:30:39.024263 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.024219    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:30:39.117415 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117380    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.117607 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117427    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m8s\" (UniqueName: \"kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.117607 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117453    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.117607 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117578    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.117721 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117633    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.117721 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.117663    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218611 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218576    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218635    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218653    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218683    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218710    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m8s\" (UniqueName: \"kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.218808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.218744    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.219155 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.219132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.219262 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.219165    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.219262 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.219208    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.221039 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.221014    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.221369 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.221347    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.227506 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.227475    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m8s\" (UniqueName: \"kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.322661 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.322567    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:39.461258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:39.461168    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:30:39.463704 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:30:39.463675    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca340db4_7c44_4b59_a6d2_a1aadb580372.slice/crio-9af3e1e1f48c8ed859b8d2f72d0ed3791a3e93d4b7c2e1dca14f7d184997b866 WatchSource:0}: Error finding container 9af3e1e1f48c8ed859b8d2f72d0ed3791a3e93d4b7c2e1dca14f7d184997b866: Status 404 returned error can't find the container with id 9af3e1e1f48c8ed859b8d2f72d0ed3791a3e93d4b7c2e1dca14f7d184997b866
Jul 31 15:30:40.210435 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:40.210386    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" event={"ID":"ca340db4-7c44-4b59-a6d2-a1aadb580372","Type":"ContainerStarted","Data":"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"}
Jul 31 15:30:40.210820 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:40.210446    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" event={"ID":"ca340db4-7c44-4b59-a6d2-a1aadb580372","Type":"ContainerStarted","Data":"9af3e1e1f48c8ed859b8d2f72d0ed3791a3e93d4b7c2e1dca14f7d184997b866"}
Jul 31 15:30:40.230325 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:40.230272    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" podStartSLOduration=2.230252206 podStartE2EDuration="2.230252206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:30:40.228484782 +0000 UTC m=+2562.386736275" watchObservedRunningTime="2026-07-31 15:30:40.230252206 +0000 UTC m=+2562.388503688"
Jul 31 15:30:40.323269 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:40.323188    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:49.322903 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:49.322865    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:49.335194 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:49.335167    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:30:50.266488 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:30:50.266458    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:32:00.129182 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.129150    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:32:00.138145 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:32:00.138108    2557 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 31 15:32:00.138145 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:32:00.138145 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:32:00.138145 ip-10-0-137-96 kubenswrapper[2557]:  > containerID="213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8" cmd=["/bin/sleep","15"]
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.138163    2557 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" message="" err=<
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:  >
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:32:00.138175    2557 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:         , stdout: , stderr: , exit code -1
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main" containerID="cri-o://213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"
Jul 31 15:32:00.138469 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.138211    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main" containerID="cri-o://213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8" gracePeriod=60
Jul 31 15:32:00.255998 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.255959    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 31 15:32:00.490802 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.490775    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:32:00.581788 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.581751    2557 generic.go:356] "Generic (PLEG): container finished" podID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerID="213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8" exitCode=0
Jul 31 15:32:00.581983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.581804    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"
Jul 31 15:32:00.581983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.581841    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" event={"ID":"ca340db4-7c44-4b59-a6d2-a1aadb580372","Type":"ContainerDied","Data":"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"}
Jul 31 15:32:00.581983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.581897    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf" event={"ID":"ca340db4-7c44-4b59-a6d2-a1aadb580372","Type":"ContainerDied","Data":"9af3e1e1f48c8ed859b8d2f72d0ed3791a3e93d4b7c2e1dca14f7d184997b866"}
Jul 31 15:32:00.581983 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.581923    2557 scope.go:117] "RemoveContainer" containerID="213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"
Jul 31 15:32:00.590400 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.590371    2557 scope.go:117] "RemoveContainer" containerID="213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"
Jul 31 15:32:00.590689 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:32:00.590665    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8\": container with ID starting with 213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8 not found: ID does not exist" containerID="213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"
Jul 31 15:32:00.590761 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.590698    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8"} err="failed to get container status \"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8\": rpc error: code = NotFound desc = could not find container \"213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8\": container with ID starting with 213424068e5cd722cb3e8b079c93d46c3f29d7b0d0253a5d89b8994cb2c589e8 not found: ID does not exist"
Jul 31 15:32:00.654139 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654101    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654139 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654147    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654402 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654175    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654402 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654210    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654402 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654233    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2m8s\" (UniqueName: \"kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654402 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654304    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir\") pod \"ca340db4-7c44-4b59-a6d2-a1aadb580372\" (UID: \"ca340db4-7c44-4b59-a6d2-a1aadb580372\") "
Jul 31 15:32:00.654632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654567    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:00.654632 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654610    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home" (OuterVolumeSpecName: "home") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:00.654756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654648    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:00.654756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654746    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.654872 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654766    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.654872 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.654780    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.656659 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.656633    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s" (OuterVolumeSpecName: "kube-api-access-q2m8s") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "kube-api-access-q2m8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:00.656778 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.656666    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:32:00.656778 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.656678    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm" (OuterVolumeSpecName: "dshm") pod "ca340db4-7c44-4b59-a6d2-a1aadb580372" (UID: "ca340db4-7c44-4b59-a6d2-a1aadb580372"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:00.755893 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.755848    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca340db4-7c44-4b59-a6d2-a1aadb580372-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.755893 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.755883    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca340db4-7c44-4b59-a6d2-a1aadb580372-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.756114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.755906    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m8s\" (UniqueName: \"kubernetes.io/projected/ca340db4-7c44-4b59-a6d2-a1aadb580372-kube-api-access-q2m8s\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:32:00.904488 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.904453    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:32:00.908410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:00.908380    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997h7bcf"]
Jul 31 15:32:02.247559 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:02.247520    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" path="/var/lib/kubelet/pods/ca340db4-7c44-4b59-a6d2-a1aadb580372/volumes"
Jul 31 15:32:09.190188 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.190154    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:32:09.190749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.190530    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main"
Jul 31 15:32:09.190749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.190542    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main"
Jul 31 15:32:09.190749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.190627    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca340db4-7c44-4b59-a6d2-a1aadb580372" containerName="main"
Jul 31 15:32:09.193846 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.193829    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.199284 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.199258    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Jul 31 15:32:09.199481 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.199309    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 31 15:32:09.199599 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.199577    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:32:09.199891 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.199869    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Jul 31 15:32:09.200346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.200322    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-9sdq9\""
Jul 31 15:32:09.200520 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.200396    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Jul 31 15:32:09.222708 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222668    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.222912 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222755    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.222912 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222796    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.222912 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222828    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvbj\" (UniqueName: \"kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.223097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222956    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.223097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.222991    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.223097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.223045    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.223222 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.223093    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.244686 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.244650    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:32:09.323867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.323825    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.323867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.323872    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvbj\" (UniqueName: \"kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.323930    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.323952    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.323988    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324029    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324058    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324090    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324485 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324485 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324395    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324541    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324639 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324614    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.324876 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.324857    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.326329 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.326311    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.326566 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.326548    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.338194 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.338169    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvbj\" (UniqueName: \"kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.504041 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.503950    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:09.666629 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:09.666605    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:32:09.668811 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:32:09.668783    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a35ac8_225a_41d6_ad91_c0246088e6f6.slice/crio-a5ec4ba58bb3cc8ea6d7ae3190ce193eb42552bab5756c9d2a976c74ea9e8158 WatchSource:0}: Error finding container a5ec4ba58bb3cc8ea6d7ae3190ce193eb42552bab5756c9d2a976c74ea9e8158: Status 404 returned error can't find the container with id a5ec4ba58bb3cc8ea6d7ae3190ce193eb42552bab5756c9d2a976c74ea9e8158
Jul 31 15:32:10.629903 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:10.629868    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerStarted","Data":"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"}
Jul 31 15:32:10.630336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:10.629922    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerStarted","Data":"a5ec4ba58bb3cc8ea6d7ae3190ce193eb42552bab5756c9d2a976c74ea9e8158"}
Jul 31 15:32:12.642591 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:12.642558    2557 generic.go:356] "Generic (PLEG): container finished" podID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerID="986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191" exitCode=0
Jul 31 15:32:12.643002 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:12.642632    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerDied","Data":"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"}
Jul 31 15:32:13.649631 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:13.649595    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerStarted","Data":"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"}
Jul 31 15:32:13.692069 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:13.692008    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podStartSLOduration=2.448269626 podStartE2EDuration="4.691989071s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.243719445s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:32:13.69148588 +0000 UTC m=+2655.849737370" watchObservedRunningTime="2026-07-31 15:32:13.691989071 +0000 UTC m=+2655.850240561"
Jul 31 15:32:14.504347 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:14.504309    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:19.504512 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:19.504474    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:32:19.506329 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:19.506300    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:32:29.504667 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:29.504621    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:32:39.504783 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:39.504739    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:32:49.505155 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:49.505108    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:32:58.387055 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:58.386970    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:32:58.396154 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:58.396131    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:32:59.505467 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:32:59.505411    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:09.504642 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:09.504593    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:19.504922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:19.504867    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:29.505312 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:29.505266    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:39.504781 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:39.504738    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:49.505076 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:49.505023    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:33:59.505124 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:33:59.505069    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:34:09.515299 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:09.515266    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:34:09.523012 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:09.522987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:34:14.106670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:14.106638    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:34:29.129779 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:29.129660    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" containerID="cri-o://d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63" gracePeriod=45
Jul 31 15:34:29.516783 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:29.516735    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": dial tcp 10.132.0.58:8000: connect: connection refused"
Jul 31 15:34:29.883726 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:29.883700    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:34:30.000593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000553    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.000776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000618    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.000776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000664    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.000776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000709    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.000776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000741    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.000776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000763    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.001054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000786    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvvbj\" (UniqueName: \"kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.001054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.000824    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache\") pod \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\" (UID: \"f3a35ac8-225a-41d6-ad91-c0246088e6f6\") "
Jul 31 15:34:30.001184 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.001159    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:34:30.001326 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.001274    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:30.001512 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.001393    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home" (OuterVolumeSpecName: "home") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:30.003034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.003004    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:34:30.003200 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.003169    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj" (OuterVolumeSpecName: "kube-api-access-rvvbj") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "kube-api-access-rvvbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:34:30.003340 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.003226    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm" (OuterVolumeSpecName: "dshm") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:30.019300 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.019256    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:30.022427 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.022390    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3a35ac8-225a-41d6-ad91-c0246088e6f6" (UID: "f3a35ac8-225a-41d6-ad91-c0246088e6f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101846    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101879    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f3a35ac8-225a-41d6-ad91-c0246088e6f6-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101889    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101897    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101905    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvbj\" (UniqueName: \"kubernetes.io/projected/f3a35ac8-225a-41d6-ad91-c0246088e6f6-kube-api-access-rvvbj\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101913    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101921    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.101936 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.101927    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a35ac8-225a-41d6-ad91-c0246088e6f6-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:34:30.312697 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.312661    2557 generic.go:356] "Generic (PLEG): container finished" podID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerID="d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63" exitCode=0
Jul 31 15:34:30.313119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.312712    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"
Jul 31 15:34:30.313119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.312730    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerDied","Data":"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"}
Jul 31 15:34:30.313119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.312773    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6" event={"ID":"f3a35ac8-225a-41d6-ad91-c0246088e6f6","Type":"ContainerDied","Data":"a5ec4ba58bb3cc8ea6d7ae3190ce193eb42552bab5756c9d2a976c74ea9e8158"}
Jul 31 15:34:30.313119 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.312793    2557 scope.go:117] "RemoveContainer" containerID="d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"
Jul 31 15:34:30.321584 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.321568    2557 scope.go:117] "RemoveContainer" containerID="986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"
Jul 31 15:34:30.329499 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.329483    2557 scope.go:117] "RemoveContainer" containerID="d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"
Jul 31 15:34:30.329762 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:34:30.329739    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63\": container with ID starting with d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63 not found: ID does not exist" containerID="d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"
Jul 31 15:34:30.329867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.329768    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63"} err="failed to get container status \"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63\": rpc error: code = NotFound desc = could not find container \"d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63\": container with ID starting with d47d1c9e1c58087e3fcefa083a1c8014bd838961d3defc24a452184d4a9cae63 not found: ID does not exist"
Jul 31 15:34:30.329867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.329786    2557 scope.go:117] "RemoveContainer" containerID="986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"
Jul 31 15:34:30.330021 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:34:30.330007    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191\": container with ID starting with 986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191 not found: ID does not exist" containerID="986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"
Jul 31 15:34:30.330069 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.330024    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191"} err="failed to get container status \"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191\": rpc error: code = NotFound desc = could not find container \"986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191\": container with ID starting with 986e187fbcab82448ca4c7098357262924e489b7e2b8d7b8f802a2f986b97191 not found: ID does not exist"
Jul 31 15:34:30.332525 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.332492    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:34:30.336264 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:30.336230    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-bdf58b9b4-qxmd6"]
Jul 31 15:34:32.247400 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:32.247364    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" path="/var/lib/kubelet/pods/f3a35ac8-225a-41d6-ad91-c0246088e6f6/volumes"
Jul 31 15:34:38.214599 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.214563    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:34:38.215103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.214902    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main"
Jul 31 15:34:38.215103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.214914    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main"
Jul 31 15:34:38.215103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.214925    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="storage-initializer"
Jul 31 15:34:38.215103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.214931    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="storage-initializer"
Jul 31 15:34:38.215103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.215002    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a35ac8-225a-41d6-ad91-c0246088e6f6" containerName="main"
Jul 31 15:34:38.217943 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.217921    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.222541 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.222509    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Jul 31 15:34:38.222715 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.222576    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:34:38.222715 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.222582    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Jul 31 15:34:38.222933 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.222919    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 31 15:34:38.223146 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.223132    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-5j4vn\""
Jul 31 15:34:38.223485 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.223467    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Jul 31 15:34:38.232737 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.232709    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:34:38.273034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.272995    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273047    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273094    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273127    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273151    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273210 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273183    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8tm\" (UniqueName: \"kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273439 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273224    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.273439 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.273277    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373802 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373763    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373831    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373853    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373868    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373886    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373905    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373922    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8tm\" (UniqueName: \"kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.373994 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.373948    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.374403 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.374274    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.374403 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.374328    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.374518 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.374412    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.374648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.374627    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.374706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.374639    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.376203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.376184    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.376527 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.376504    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.382488 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.382461    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8tm\" (UniqueName: \"kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm\") pod \"custom-route-timeout-test-kserve-98f8d95f7-pf99s\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.529709 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.529617    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:38.682610 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:38.682582    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:34:38.685149 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:34:38.685118    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b7d2c1_2c30_48c1_aad1_29c05b6c38a9.slice/crio-e50b1f13de2383cfab3bf84b60b21245c21d788fc271c7f7e24ac5d0747165f4 WatchSource:0}: Error finding container e50b1f13de2383cfab3bf84b60b21245c21d788fc271c7f7e24ac5d0747165f4: Status 404 returned error can't find the container with id e50b1f13de2383cfab3bf84b60b21245c21d788fc271c7f7e24ac5d0747165f4
Jul 31 15:34:39.360168 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:39.360132    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerStarted","Data":"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"}
Jul 31 15:34:39.360707 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:39.360182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerStarted","Data":"e50b1f13de2383cfab3bf84b60b21245c21d788fc271c7f7e24ac5d0747165f4"}
Jul 31 15:34:41.370575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:41.370482    2557 generic.go:356] "Generic (PLEG): container finished" podID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerID="42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb" exitCode=0
Jul 31 15:34:41.370575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:41.370561    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerDied","Data":"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"}
Jul 31 15:34:42.378754 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:42.378721    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerStarted","Data":"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"}
Jul 31 15:34:42.404074 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:42.404017    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podStartSLOduration=2.180826577 podStartE2EDuration="4.403999182s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.223172605s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:34:42.402489303 +0000 UTC m=+2804.560740793" watchObservedRunningTime="2026-07-31 15:34:42.403999182 +0000 UTC m=+2804.562250671"
Jul 31 15:34:42.529943 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:42.529906    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:48.530378 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:48.530344    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:34:48.531924 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:48.531897    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:34:58.530225 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:34:58.530176    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:08.530548 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:08.530502    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:18.530338 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:18.530292    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:28.530929 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:28.530877    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:38.530677 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:38.530624    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:48.530497 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:48.530456    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:35:58.530971 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:35:58.530873    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:36:08.531089 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:08.531037    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:36:18.530354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:18.530306    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:36:28.530679 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:28.530639    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:36:38.540228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:38.540195    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:36:38.548012 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:38.547987    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:36:43.422217 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:43.422179    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:36:58.444398 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:58.444328    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" containerID="cri-o://43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f" gracePeriod=45
Jul 31 15:36:58.541453 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:58.541413    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 31 15:36:59.203034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.203010    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:36:59.261276 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261169    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261276 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261214    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261276 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261261    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261588 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261286    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px8tm\" (UniqueName: \"kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261588 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261350    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261588 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261378    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261588 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261398    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261588 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261424    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir\") pod \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\" (UID: \"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9\") "
Jul 31 15:36:59.261844 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261582    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:59.261844 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261671    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:36:59.261844 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261782    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.261844 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.261803    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.262132 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.262107    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home" (OuterVolumeSpecName: "home") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:59.263722 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.263696    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:36:59.263905 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.263887    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm" (OuterVolumeSpecName: "dshm") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:59.263969 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.263906    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm" (OuterVolumeSpecName: "kube-api-access-px8tm") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "kube-api-access-px8tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:36:59.280659 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.280621    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:59.290987 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.290958    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" (UID: "f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:59.362346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362311    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.362346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362337    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.362346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362347    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.362602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362373    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.362602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362383    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:36:59.362602 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:36:59.362393    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8tm\" (UniqueName: \"kubernetes.io/projected/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9-kube-api-access-px8tm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:37:00.049861 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.049824    2557 generic.go:356] "Generic (PLEG): container finished" podID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerID="43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f" exitCode=0
Jul 31 15:37:00.050311 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.049883    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"
Jul 31 15:37:00.050311 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.049915    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerDied","Data":"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"}
Jul 31 15:37:00.050311 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.049977    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s" event={"ID":"f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9","Type":"ContainerDied","Data":"e50b1f13de2383cfab3bf84b60b21245c21d788fc271c7f7e24ac5d0747165f4"}
Jul 31 15:37:00.050311 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.050010    2557 scope.go:117] "RemoveContainer" containerID="43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"
Jul 31 15:37:00.062671 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.061300    2557 scope.go:117] "RemoveContainer" containerID="42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"
Jul 31 15:37:00.074141 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.074118    2557 scope.go:117] "RemoveContainer" containerID="43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"
Jul 31 15:37:00.074521 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:37:00.074496    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f\": container with ID starting with 43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f not found: ID does not exist" containerID="43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"
Jul 31 15:37:00.074625 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.074538    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f"} err="failed to get container status \"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f\": rpc error: code = NotFound desc = could not find container \"43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f\": container with ID starting with 43e328414763563be21dfa8439435185987ec9cdd4f862baacec835393e6197f not found: ID does not exist"
Jul 31 15:37:00.074625 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.074569    2557 scope.go:117] "RemoveContainer" containerID="42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"
Jul 31 15:37:00.074849 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:37:00.074831    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb\": container with ID starting with 42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb not found: ID does not exist" containerID="42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"
Jul 31 15:37:00.074889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.074856    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb"} err="failed to get container status \"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb\": rpc error: code = NotFound desc = could not find container \"42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb\": container with ID starting with 42cdb8ca9d7f0370adad6aef2c3db3ce04fafc5be068667c7c77d95f52d52ddb not found: ID does not exist"
Jul 31 15:37:00.078338 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.078310    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:37:00.081715 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.081691    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-98f8d95f7-pf99s"]
Jul 31 15:37:00.247515 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:00.247481    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" path="/var/lib/kubelet/pods/f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9/volumes"
Jul 31 15:37:25.222307 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.222071    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:37:25.223689 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.223660    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="storage-initializer"
Jul 31 15:37:25.224537 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.224509    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="storage-initializer"
Jul 31 15:37:25.224749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.224726    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main"
Jul 31 15:37:25.224749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.224750    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main"
Jul 31 15:37:25.224857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.224838    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b7d2c1-2c30-48c1-aad1-29c05b6c38a9" containerName="main"
Jul 31 15:37:25.228068 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.228049    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.230433 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230406    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 31 15:37:25.230557 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230461    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Jul 31 15:37:25.230557 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230515    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Jul 31 15:37:25.230646 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230629    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:37:25.230697 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230643    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-8nk2s\""
Jul 31 15:37:25.230697 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.230666    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Jul 31 15:37:25.234035 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.234014    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:37:25.291531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291541    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291797 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291587    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291797 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291644    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291797 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291673    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291797 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291767    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291960 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291800    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.291960 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.291820    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h27\" (UniqueName: \"kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392478 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392444    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392491    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392551    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392577    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392642    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392674    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392971 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392699    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h27\" (UniqueName: \"kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392971 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392746    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.392971 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392958    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.393159 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.392988    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.393213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.393175    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.393282 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.393217    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.393461 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.393444    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.394966 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.394950    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.395246 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.395215    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.404527 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.404501    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h27\" (UniqueName: \"kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27\") pod \"router-with-refs-test-kserve-5b5565f6cc-t58xb\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.540332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.540223    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:25.692638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:25.692605    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:37:25.696621 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:37:25.696592    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd933679f_d1d3_47bd_9225_820567ed6973.slice/crio-729b741b296a02cff55fb756568efa7bf4a8877a3bbb76eba3616d6dd566e02d WatchSource:0}: Error finding container 729b741b296a02cff55fb756568efa7bf4a8877a3bbb76eba3616d6dd566e02d: Status 404 returned error can't find the container with id 729b741b296a02cff55fb756568efa7bf4a8877a3bbb76eba3616d6dd566e02d
Jul 31 15:37:26.177578 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:26.177534    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerStarted","Data":"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"}
Jul 31 15:37:26.177788 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:26.177594    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerStarted","Data":"729b741b296a02cff55fb756568efa7bf4a8877a3bbb76eba3616d6dd566e02d"}
Jul 31 15:37:28.192329 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:28.192295    2557 generic.go:356] "Generic (PLEG): container finished" podID="d933679f-d1d3-47bd-9225-820567ed6973" containerID="cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5" exitCode=0
Jul 31 15:37:28.192747 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:28.192372    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerDied","Data":"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"}
Jul 31 15:37:29.202956 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:29.202908    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerStarted","Data":"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"}
Jul 31 15:37:29.229356 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:29.229303    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podStartSLOduration=1.970296973 podStartE2EDuration="4.22928698s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.258990007s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:37:29.22650092 +0000 UTC m=+2971.384752411" watchObservedRunningTime="2026-07-31 15:37:29.22928698 +0000 UTC m=+2971.387538470"
Jul 31 15:37:29.541426 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:29.541326    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:35.540799 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:35.540760    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:37:35.542394 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:35.542362    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:37:45.541478 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:45.541434    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:37:55.541404 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:55.541356    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:37:58.417816 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:58.417786    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:37:58.427384 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:37:58.427348    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:38:05.541615 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:05.541570    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:38:15.540957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:15.540912    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:38:25.541002 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:25.540958    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:38:35.541097 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:35.541053    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:38:45.540734 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:45.540684    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:38:55.541344 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:38:55.541248    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:39:05.541536 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:05.541490    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:39:15.540965 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:15.540915    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:39:25.541576 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:25.541522    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": dial tcp 10.132.0.60:8000: connect: connection refused"
Jul 31 15:39:35.551762 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:35.551728    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:39:35.560063 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:35.560029    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:39:40.693283 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:40.693228    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:39:55.715268 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:55.715205    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main" containerID="cri-o://00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744" gracePeriod=45
Jul 31 15:39:56.472011 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.471980    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:39:56.607875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.607781    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.607875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.607834    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.607875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.607857    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5h27\" (UniqueName: \"kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.607884    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608171 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608056    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608181    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:39:56.608320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608193    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608224    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608320 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608284    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir\") pod \"d933679f-d1d3-47bd-9225-820567ed6973\" (UID: \"d933679f-d1d3-47bd-9225-820567ed6973\") "
Jul 31 15:39:56.608514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608350    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache" (OuterVolumeSpecName: "model-cache") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:56.608642 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608589    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d933679f-d1d3-47bd-9225-820567ed6973-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.608642 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608613    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.608780 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.608665    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home" (OuterVolumeSpecName: "home") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:56.610353 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.610323    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:39:56.610592 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.610570    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm" (OuterVolumeSpecName: "dshm") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:56.610792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.610766    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27" (OuterVolumeSpecName: "kube-api-access-w5h27") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "kube-api-access-w5h27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:39:56.621187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.621158    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:56.638420 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.638362    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d933679f-d1d3-47bd-9225-820567ed6973" (UID: "d933679f-d1d3-47bd-9225-820567ed6973"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:56.709268 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709202    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5h27\" (UniqueName: \"kubernetes.io/projected/d933679f-d1d3-47bd-9225-820567ed6973-kube-api-access-w5h27\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.709268 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709265    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.709494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709282    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d933679f-d1d3-47bd-9225-820567ed6973-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.709494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709296    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.709494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709307    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.709494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.709318    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d933679f-d1d3-47bd-9225-820567ed6973-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:39:56.945394 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.945351    2557 generic.go:356] "Generic (PLEG): container finished" podID="d933679f-d1d3-47bd-9225-820567ed6973" containerID="00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744" exitCode=0
Jul 31 15:39:56.945902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.945411    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"
Jul 31 15:39:56.945902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.945438    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerDied","Data":"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"}
Jul 31 15:39:56.945902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.945547    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb" event={"ID":"d933679f-d1d3-47bd-9225-820567ed6973","Type":"ContainerDied","Data":"729b741b296a02cff55fb756568efa7bf4a8877a3bbb76eba3616d6dd566e02d"}
Jul 31 15:39:56.945902 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.945582    2557 scope.go:117] "RemoveContainer" containerID="00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"
Jul 31 15:39:56.955859 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.955837    2557 scope.go:117] "RemoveContainer" containerID="cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"
Jul 31 15:39:56.963773 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.963754    2557 scope.go:117] "RemoveContainer" containerID="00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"
Jul 31 15:39:56.964075 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:39:56.964035    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744\": container with ID starting with 00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744 not found: ID does not exist" containerID="00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"
Jul 31 15:39:56.964153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.964084    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744"} err="failed to get container status \"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744\": rpc error: code = NotFound desc = could not find container \"00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744\": container with ID starting with 00dd4f8a552a9d61bddb03d9779c5564d9225a73469c381bf87c7ffcac44f744 not found: ID does not exist"
Jul 31 15:39:56.964153 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.964103    2557 scope.go:117] "RemoveContainer" containerID="cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"
Jul 31 15:39:56.964381 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:39:56.964362    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5\": container with ID starting with cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5 not found: ID does not exist" containerID="cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"
Jul 31 15:39:56.964433 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.964385    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5"} err="failed to get container status \"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5\": rpc error: code = NotFound desc = could not find container \"cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5\": container with ID starting with cc60576438c7d156cd6323b3ee281b940354695944c9a4502336b1856aaf62e5 not found: ID does not exist"
Jul 31 15:39:56.968838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.968814    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:39:56.972517 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:56.972495    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b5565f6cc-t58xb"]
Jul 31 15:39:58.249225 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:39:58.249192    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d933679f-d1d3-47bd-9225-820567ed6973" path="/var/lib/kubelet/pods/d933679f-d1d3-47bd-9225-820567ed6973/volumes"
Jul 31 15:40:17.768751 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.768714    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:40:17.769206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.769058    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="storage-initializer"
Jul 31 15:40:17.769206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.769069    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="storage-initializer"
Jul 31 15:40:17.769206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.769086    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main"
Jul 31 15:40:17.769206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.769091    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main"
Jul 31 15:40:17.769206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.769169    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="d933679f-d1d3-47bd-9225-820567ed6973" containerName="main"
Jul 31 15:40:17.771838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.771818    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.774448 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.774394    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Jul 31 15:40:17.774448 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.774429    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-z7mr6\""
Jul 31 15:40:17.774659 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.774512    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Jul 31 15:40:17.774786 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.774766    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-6d5jn\""
Jul 31 15:40:17.775446 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.775428    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:40:17.775549 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.775464    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Jul 31 15:40:17.775638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.775616    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 31 15:40:17.787724 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.787696    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:40:17.899076 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899038    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfp7w\" (UniqueName: \"kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899076 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899079    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899148    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899166    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899231    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899311    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899336 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899333    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:17.899505 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:17.899348    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000575 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000538    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000765 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000584    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000765 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000615    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000765 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000731    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000887 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000764    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000887 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000855    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp7w\" (UniqueName: \"kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.000970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000897    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.000976    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.001015    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001127 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.001073    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001181 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.001121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001295 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.001268    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.001464 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.001446    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.003060 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.003035    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.003354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.003336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.009605 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.009579    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp7w\" (UniqueName: \"kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.082721 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.082623    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:18.229890 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:18.229859    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:40:18.231407 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:40:18.231381    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef3377e_4680_4b17_8cbc_5e597848b1cc.slice/crio-3d2e36c8193f5fca345e2571a839b64a03979d04ab86fde7ae8a08539df76738 WatchSource:0}: Error finding container 3d2e36c8193f5fca345e2571a839b64a03979d04ab86fde7ae8a08539df76738: Status 404 returned error can't find the container with id 3d2e36c8193f5fca345e2571a839b64a03979d04ab86fde7ae8a08539df76738
Jul 31 15:40:19.054176 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:19.054138    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerStarted","Data":"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"}
Jul 31 15:40:19.054643 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:19.054197    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerStarted","Data":"3d2e36c8193f5fca345e2571a839b64a03979d04ab86fde7ae8a08539df76738"}
Jul 31 15:40:19.054643 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:19.054229    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:20.061127 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:20.061047    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerStarted","Data":"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"}
Jul 31 15:40:22.073436 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:22.073397    2557 generic.go:356] "Generic (PLEG): container finished" podID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerID="3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388" exitCode=0
Jul 31 15:40:22.073881 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:22.073475    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerDied","Data":"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"}
Jul 31 15:40:23.080228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:23.080185    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerStarted","Data":"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"}
Jul 31 15:40:23.106194 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:23.106132    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podStartSLOduration=3.873507548 podStartE2EDuration="6.106115063s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.232607515s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:40:23.103582691 +0000 UTC m=+3145.261834181" watchObservedRunningTime="2026-07-31 15:40:23.106115063 +0000 UTC m=+3145.264366553"
Jul 31 15:40:28.083621 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:28.083581    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:28.084309 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:28.083661    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:28.085121 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:28.085081    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:40:28.097684 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:28.097646    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:40:38.083198 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:38.083146    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:40:48.083539 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:48.083494    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:40:58.083970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:40:58.083917    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:08.083534 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:08.083483    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:18.083228 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:18.083180    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:28.083447 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:28.083396    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:38.083937 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:38.083882    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:48.083306 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:48.083212    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:41:58.083226 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:41:58.083069    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:42:08.083362 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:08.083312    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:42:18.084032 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:18.083982    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 31 15:42:28.094027 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:28.093989    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:42:28.110648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:28.110621    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:42:36.905736 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:36.905697    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:42:51.928695 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:51.928652    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main" containerID="cri-o://599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe" gracePeriod=45
Jul 31 15:42:52.546908 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.546871    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="llm-d-routing-sidecar" containerID="cri-o://5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498" gracePeriod=45
Jul 31 15:42:52.794067 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.794037    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:42:52.817501 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817463    2557 generic.go:356] "Generic (PLEG): container finished" podID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerID="599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe" exitCode=0
Jul 31 15:42:52.817501 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817498    2557 generic.go:356] "Generic (PLEG): container finished" podID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerID="5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498" exitCode=0
Jul 31 15:42:52.817741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817516    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"
Jul 31 15:42:52.817741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817551    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerDied","Data":"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"}
Jul 31 15:42:52.817741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817612    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerDied","Data":"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"}
Jul 31 15:42:52.817741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817630    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz" event={"ID":"cef3377e-4680-4b17-8cbc-5e597848b1cc","Type":"ContainerDied","Data":"3d2e36c8193f5fca345e2571a839b64a03979d04ab86fde7ae8a08539df76738"}
Jul 31 15:42:52.817741 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.817653    2557 scope.go:117] "RemoveContainer" containerID="599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"
Jul 31 15:42:52.828346 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.828321    2557 scope.go:117] "RemoveContainer" containerID="3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"
Jul 31 15:42:52.837278 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.837255    2557 scope.go:117] "RemoveContainer" containerID="5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"
Jul 31 15:42:52.845666 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.845644    2557 scope.go:117] "RemoveContainer" containerID="599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"
Jul 31 15:42:52.846020 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:42:52.845997    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe\": container with ID starting with 599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe not found: ID does not exist" containerID="599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"
Jul 31 15:42:52.846068 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846030    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"} err="failed to get container status \"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe\": rpc error: code = NotFound desc = could not find container \"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe\": container with ID starting with 599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe not found: ID does not exist"
Jul 31 15:42:52.846068 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846051    2557 scope.go:117] "RemoveContainer" containerID="3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"
Jul 31 15:42:52.846334 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:42:52.846317    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388\": container with ID starting with 3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388 not found: ID does not exist" containerID="3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"
Jul 31 15:42:52.846382 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846340    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"} err="failed to get container status \"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388\": rpc error: code = NotFound desc = could not find container \"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388\": container with ID starting with 3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388 not found: ID does not exist"
Jul 31 15:42:52.846382 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846356    2557 scope.go:117] "RemoveContainer" containerID="5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"
Jul 31 15:42:52.846604 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:42:52.846587    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498\": container with ID starting with 5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498 not found: ID does not exist" containerID="5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"
Jul 31 15:42:52.846650 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846608    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"} err="failed to get container status \"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498\": rpc error: code = NotFound desc = could not find container \"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498\": container with ID starting with 5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498 not found: ID does not exist"
Jul 31 15:42:52.846650 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846623    2557 scope.go:117] "RemoveContainer" containerID="599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"
Jul 31 15:42:52.846861 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846843    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe"} err="failed to get container status \"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe\": rpc error: code = NotFound desc = could not find container \"599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe\": container with ID starting with 599f481975e31a6caeefe7c65e709bcb4b4ddd73e5460dbeef9bfbad3b7218fe not found: ID does not exist"
Jul 31 15:42:52.846911 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.846862    2557 scope.go:117] "RemoveContainer" containerID="3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"
Jul 31 15:42:52.847079 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.847063    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388"} err="failed to get container status \"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388\": rpc error: code = NotFound desc = could not find container \"3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388\": container with ID starting with 3828e6cf635c8e2c14efb949e4501dd3f4dc9455e7da745280737c2ae205e388 not found: ID does not exist"
Jul 31 15:42:52.847128 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.847079    2557 scope.go:117] "RemoveContainer" containerID="5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"
Jul 31 15:42:52.847387 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.847364    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498"} err="failed to get container status \"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498\": rpc error: code = NotFound desc = could not find container \"5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498\": container with ID starting with 5ec9755519462b407a2322ebd48636ed58db35e974cae1405d19c741e631f498 not found: ID does not exist"
Jul 31 15:42:52.956600 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956566    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956612    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956643    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956677    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956702    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956763    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956810    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfp7w\" (UniqueName: \"kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.956832    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert\") pod \"cef3377e-4680-4b17-8cbc-5e597848b1cc\" (UID: \"cef3377e-4680-4b17-8cbc-5e597848b1cc\") "
Jul 31 15:42:52.957051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.957030    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.957611 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.957155    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:52.957611 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.957404    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home" (OuterVolumeSpecName: "home") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.957611 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.957450    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:42:52.959481 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.959453    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:42:52.959601 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.959481    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm" (OuterVolumeSpecName: "dshm") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.959647 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.959600    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w" (OuterVolumeSpecName: "kube-api-access-rfp7w") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "kube-api-access-rfp7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:42:52.975159 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.975128    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:52.982186 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:52.982155    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cef3377e-4680-4b17-8cbc-5e597848b1cc" (UID: "cef3377e-4680-4b17-8cbc-5e597848b1cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:53.058315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058282    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058310    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058320    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058315 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058330    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef3377e-4680-4b17-8cbc-5e597848b1cc-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058608 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058339    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef3377e-4680-4b17-8cbc-5e597848b1cc-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058608 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058349    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfp7w\" (UniqueName: \"kubernetes.io/projected/cef3377e-4680-4b17-8cbc-5e597848b1cc-kube-api-access-rfp7w\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.058608 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.058357    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cef3377e-4680-4b17-8cbc-5e597848b1cc-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:42:53.143477 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.143446    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:42:53.155514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:53.152568    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7d5fd4bfb6n62lz"]
Jul 31 15:42:54.247596 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:54.247556    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" path="/var/lib/kubelet/pods/cef3377e-4680-4b17-8cbc-5e597848b1cc/volumes"
Jul 31 15:42:58.446867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:58.446834    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:42:58.458661 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:42:58.458629    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:43:08.163412 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163375    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163772    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="llm-d-routing-sidecar"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163796    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="llm-d-routing-sidecar"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163809    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="storage-initializer"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163814    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="storage-initializer"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163840    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163849    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main"
Jul 31 15:43:08.163935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163934    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="main"
Jul 31 15:43:08.164321 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.163943    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef3377e-4680-4b17-8cbc-5e597848b1cc" containerName="llm-d-routing-sidecar"
Jul 31 15:43:08.169463 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.169438    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.172078 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172049    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 31 15:43:08.172217 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172163    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-lvlsf\""
Jul 31 15:43:08.172309 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172269    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 31 15:43:08.172358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172301    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 31 15:43:08.172358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172280    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:43:08.172760 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172743    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-bmxgl\""
Jul 31 15:43:08.172825 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.172781    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 31 15:43:08.184369 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.184334    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:43:08.202875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.202844    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203033 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.202880    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203033 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.202905    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvvbk\" (UniqueName: \"kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203033 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.202984    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203033 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.203025    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203277 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.203049    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203277 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.203105    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.203277 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.203141    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304289 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304250    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304319    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304337    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304358    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304388    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304497    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304530    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304621    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvvbk\" (UniqueName: \"kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304693 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304675    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304960 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304722    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.304960 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.304862    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.305226 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.305207    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.306702 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.306676    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.307138 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.307119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.313776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.313743    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvvbk\" (UniqueName: \"kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk\") pod \"custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.480502 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.480466    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:08.622581 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.622554    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:43:08.625231 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:43:08.625199    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe8d6b7_19a3_480e_a948_7bdd418b9bea.slice/crio-57316959e56b916c25ab69cbd7e71d6c58105118a4606ecb727cc30d96a4d49f WatchSource:0}: Error finding container 57316959e56b916c25ab69cbd7e71d6c58105118a4606ecb727cc30d96a4d49f: Status 404 returned error can't find the container with id 57316959e56b916c25ab69cbd7e71d6c58105118a4606ecb727cc30d96a4d49f
Jul 31 15:43:08.893522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.893425    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerStarted","Data":"a9d890f9852672354a6fe9d88c1fec92f160e6be6c2888649a6d4ccd6abcbff5"}
Jul 31 15:43:08.893522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.893481    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerStarted","Data":"57316959e56b916c25ab69cbd7e71d6c58105118a4606ecb727cc30d96a4d49f"}
Jul 31 15:43:08.893522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:08.893515    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:09.900012 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:09.899978    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerStarted","Data":"9b8776901c44b5384d92044ae4591e156fd693fb312e859512cc5b7551dbadd0"}
Jul 31 15:43:11.913540 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:11.913501    2557 generic.go:356] "Generic (PLEG): container finished" podID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerID="9b8776901c44b5384d92044ae4591e156fd693fb312e859512cc5b7551dbadd0" exitCode=0
Jul 31 15:43:11.913968 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:11.913567    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerDied","Data":"9b8776901c44b5384d92044ae4591e156fd693fb312e859512cc5b7551dbadd0"}
Jul 31 15:43:12.924838 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:12.924801    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerStarted","Data":"8b93f704424bc7b0ee56e397b19f2b84f431f0b66c1ec3d90e3c1130399260c2"}
Jul 31 15:43:12.950227 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:12.950174    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podStartSLOduration=2.725812219 podStartE2EDuration="4.950144844s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.224332625s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:43:12.947541688 +0000 UTC m=+3315.105793175" watchObservedRunningTime="2026-07-31 15:43:12.950144844 +0000 UTC m=+3315.108396333"
Jul 31 15:43:18.480947 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:18.480903    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:18.481478 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:18.480990    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:18.482377 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:18.482336    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:43:18.496256 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:18.496215    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:43:28.481739 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:28.481632    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:43:38.481070 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:38.481024    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:43:48.481846 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:48.481787    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:43:58.481118 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:43:58.481056    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:08.481874 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:08.481828    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:18.481731 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:18.481685    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:28.481364 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:28.481293    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:38.481987 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:38.481929    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:48.481714 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:48.481666    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:44:58.481819 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:44:58.481648    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:45:08.481457 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:08.481380    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8001/health\": dial tcp 10.132.0.62:8001: connect: connection refused"
Jul 31 15:45:18.491600 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:18.491564    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:45:18.507447 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:18.507414    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:45:27.814317 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:27.814227    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:45:42.839584 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:42.839518    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main" containerID="cri-o://8b93f704424bc7b0ee56e397b19f2b84f431f0b66c1ec3d90e3c1130399260c2" gracePeriod=45
Jul 31 15:45:43.468648 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.468590    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="llm-d-routing-sidecar" containerID="cri-o://a9d890f9852672354a6fe9d88c1fec92f160e6be6c2888649a6d4ccd6abcbff5" gracePeriod=45
Jul 31 15:45:43.709673 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.709642    2557 generic.go:356] "Generic (PLEG): container finished" podID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerID="8b93f704424bc7b0ee56e397b19f2b84f431f0b66c1ec3d90e3c1130399260c2" exitCode=0
Jul 31 15:45:43.709673 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.709669    2557 generic.go:356] "Generic (PLEG): container finished" podID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerID="a9d890f9852672354a6fe9d88c1fec92f160e6be6c2888649a6d4ccd6abcbff5" exitCode=0
Jul 31 15:45:43.709840 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.709698    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerDied","Data":"8b93f704424bc7b0ee56e397b19f2b84f431f0b66c1ec3d90e3c1130399260c2"}
Jul 31 15:45:43.709840 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.709738    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerDied","Data":"a9d890f9852672354a6fe9d88c1fec92f160e6be6c2888649a6d4ccd6abcbff5"}
Jul 31 15:45:43.731026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.730993    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:45:43.791763 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791718    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.791763 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791768    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791799    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791853    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791872    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791917    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.791958    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792002    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvvbk\" (UniqueName: \"kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk\") pod \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\" (UID: \"7fe8d6b7-19a3-480e-a948-7bdd418b9bea\") "
Jul 31 15:45:43.792421 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792123    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.792421 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792193    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:45:43.792421 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792354    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.792421 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792402    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.792875 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.792853    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home" (OuterVolumeSpecName: "home") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.794459 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.794426    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm" (OuterVolumeSpecName: "dshm") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.794662 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.794639    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:45:43.794779 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.794760    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk" (OuterVolumeSpecName: "kube-api-access-tvvbk") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "kube-api-access-tvvbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:45:43.810177 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.810137    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.818193 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.818141    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fe8d6b7-19a3-480e-a948-7bdd418b9bea" (UID: "7fe8d6b7-19a3-480e-a948-7bdd418b9bea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:45:43.893672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893633    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.893672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893667    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.893672 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893677    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.894110 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893686    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.894110 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893695    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvvbk\" (UniqueName: \"kubernetes.io/projected/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-kube-api-access-tvvbk\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:43.894110 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:43.893702    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fe8d6b7-19a3-480e-a948-7bdd418b9bea-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:45:44.716604 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.716563    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr" event={"ID":"7fe8d6b7-19a3-480e-a948-7bdd418b9bea","Type":"ContainerDied","Data":"57316959e56b916c25ab69cbd7e71d6c58105118a4606ecb727cc30d96a4d49f"}
Jul 31 15:45:44.716604 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.716586    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"
Jul 31 15:45:44.716876 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.716638    2557 scope.go:117] "RemoveContainer" containerID="8b93f704424bc7b0ee56e397b19f2b84f431f0b66c1ec3d90e3c1130399260c2"
Jul 31 15:45:44.725862 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.725840    2557 scope.go:117] "RemoveContainer" containerID="9b8776901c44b5384d92044ae4591e156fd693fb312e859512cc5b7551dbadd0"
Jul 31 15:45:44.734987 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.734959    2557 scope.go:117] "RemoveContainer" containerID="a9d890f9852672354a6fe9d88c1fec92f160e6be6c2888649a6d4ccd6abcbff5"
Jul 31 15:45:44.737357 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.737327    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:45:44.741781 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:44.741755    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-57946ff4b7-gfchr"]
Jul 31 15:45:46.247742 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:45:46.247704    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" path="/var/lib/kubelet/pods/7fe8d6b7-19a3-480e-a948-7bdd418b9bea/volumes"
Jul 31 15:46:02.853264 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853216    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:46:02.853772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853720    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="llm-d-routing-sidecar"
Jul 31 15:46:02.853772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853738    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="llm-d-routing-sidecar"
Jul 31 15:46:02.853772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853771    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main"
Jul 31 15:46:02.853946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853779    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main"
Jul 31 15:46:02.853946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853803    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="storage-initializer"
Jul 31 15:46:02.853946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853812    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="storage-initializer"
Jul 31 15:46:02.853946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853896    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="main"
Jul 31 15:46:02.853946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.853909    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe8d6b7-19a3-480e-a948-7bdd418b9bea" containerName="llm-d-routing-sidecar"
Jul 31 15:46:02.865756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.865728    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.868551 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868478    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 31 15:46:02.868551 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868478    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 31 15:46:02.868790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868478    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 31 15:46:02.868790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868478    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:46:02.868790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868705    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 31 15:46:02.868790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.868728    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-klgcq\""
Jul 31 15:46:02.869354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.869334    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-z9ljw\""
Jul 31 15:46:02.875410 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.875383    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:46:02.974265 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974276    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974302    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974316    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974345    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974424    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974462 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974460    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:02.974670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:02.974482    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrqqh\" (UniqueName: \"kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075609 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075561    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075609 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075612    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075628    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075659    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075692    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075748    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrqqh\" (UniqueName: \"kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.075889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.075792    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.076093 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.076010    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.076451 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.076421    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.076451 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.076445    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.076608 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.076462    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.076608 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.076503    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.078372 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.078336    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.078494 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.078405    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.084614 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.084591    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrqqh\" (UniqueName: \"kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh\") pod \"router-with-refs-pd-test-kserve-9c5689f55-hq66n\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.178572 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.178537    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:03.332664 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.332620    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:46:03.335183 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:46:03.335153    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727037e0_11aa_4d70_890a_6a208351af1b.slice/crio-7a287b504be6a8eb001b7a58c0615665a31477db5711e646ea4a185127858366 WatchSource:0}: Error finding container 7a287b504be6a8eb001b7a58c0615665a31477db5711e646ea4a185127858366: Status 404 returned error can't find the container with id 7a287b504be6a8eb001b7a58c0615665a31477db5711e646ea4a185127858366
Jul 31 15:46:03.808834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.808785    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerStarted","Data":"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"}
Jul 31 15:46:03.809048 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.808863    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerStarted","Data":"7a287b504be6a8eb001b7a58c0615665a31477db5711e646ea4a185127858366"}
Jul 31 15:46:03.809048 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:03.808911    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:04.816573 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:04.816541    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerStarted","Data":"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"}
Jul 31 15:46:06.828074 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:06.828036    2557 generic.go:356] "Generic (PLEG): container finished" podID="727037e0-11aa-4d70-890a-6a208351af1b" containerID="0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b" exitCode=0
Jul 31 15:46:06.828522 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:06.828102    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerDied","Data":"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"}
Jul 31 15:46:07.835867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:07.835828    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerStarted","Data":"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"}
Jul 31 15:46:07.862691 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:07.862623    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podStartSLOduration=3.409545762 podStartE2EDuration="5.86259914s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.453053378s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:46:07.860304863 +0000 UTC m=+3490.018556353" watchObservedRunningTime="2026-07-31 15:46:07.86259914 +0000 UTC m=+3490.020850632"
Jul 31 15:46:13.179224 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:13.179175    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:13.180008 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:13.179275    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:13.180670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:13.180635    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:46:13.197343 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:13.197307    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:46:23.180059 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:23.179933    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:46:33.179990 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:33.179941    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:46:34.488156 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.488114    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:46:34.492078 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.492049    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.494717 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494682    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 31 15:46:34.494900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494804    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 31 15:46:34.494900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494820    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 31 15:46:34.494900 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494857    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:46:34.495089 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494920    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-m8lw2\""
Jul 31 15:46:34.495089 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.494924    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 31 15:46:34.503081 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.503049    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:46:34.586259 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586205    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnj69\" (UniqueName: \"kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586510 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586332    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586510 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586379    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586510 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586496    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586678 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586526    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586678 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586572    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586759 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586664    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.586759 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.586738    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687197 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687155    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnj69\" (UniqueName: \"kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687427 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687206    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687427 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687366    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687564 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687489    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687564 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687519    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687683 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687562    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687683 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687606    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687683 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687638    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687683 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687668    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.687890 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.687817    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.688150 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.688121    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.688326 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.688132    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.688326 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.688227    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.690147 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.690115    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.690393 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.690370    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.696516 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.696483    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnj69\" (UniqueName: \"kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69\") pod \"prestop-hook-test-kserve-56fc49c9b5-6z45b\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.805008 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.804854    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:34.967992 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.967956    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:46:34.972588 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:46:34.972555    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79323635_0922_47c8_8966_12c7f847dd52.slice/crio-dfde94f3bf7d134050ad559a4d800f32a060a4135522d7dbcd05e23884bf3c51 WatchSource:0}: Error finding container dfde94f3bf7d134050ad559a4d800f32a060a4135522d7dbcd05e23884bf3c51: Status 404 returned error can't find the container with id dfde94f3bf7d134050ad559a4d800f32a060a4135522d7dbcd05e23884bf3c51
Jul 31 15:46:34.986558 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:34.986523    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerStarted","Data":"dfde94f3bf7d134050ad559a4d800f32a060a4135522d7dbcd05e23884bf3c51"}
Jul 31 15:46:35.996349 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:35.996288    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerStarted","Data":"64a73f1767c97e0e0f08d75f5f29b024bb1d0c68587d8a310eb8403745b6210d"}
Jul 31 15:46:38.014651 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:38.014597    2557 generic.go:356] "Generic (PLEG): container finished" podID="79323635-0922-47c8-8966-12c7f847dd52" containerID="64a73f1767c97e0e0f08d75f5f29b024bb1d0c68587d8a310eb8403745b6210d" exitCode=0
Jul 31 15:46:38.015338 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:38.014673    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerDied","Data":"64a73f1767c97e0e0f08d75f5f29b024bb1d0c68587d8a310eb8403745b6210d"}
Jul 31 15:46:39.023646 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:39.023588    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerStarted","Data":"8f0942c7421a9c634f9c03e0b583df1c250b9163ecab249351adfb2bcc5fbc5a"}
Jul 31 15:46:39.051490 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:39.051422    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" podStartSLOduration=2.484485564 podStartE2EDuration="5.051402228s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.566916664s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:46:39.05045765 +0000 UTC m=+3521.208709140" watchObservedRunningTime="2026-07-31 15:46:39.051402228 +0000 UTC m=+3521.209653718"
Jul 31 15:46:39.805420 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:39.805380    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:43.179948 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:43.179894    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:46:44.805797 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:44.805746    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:46:44.807647 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:44.807608    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 31 15:46:52.800283 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:52.800225    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:46:53.179701 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:46:53.179657    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:03.179746 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:03.179695    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:07.827815 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:07.827769    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="main" containerID="cri-o://8f0942c7421a9c634f9c03e0b583df1c250b9163ecab249351adfb2bcc5fbc5a" gracePeriod=45
Jul 31 15:47:13.179476 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:13.179422    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:23.180109 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:23.180043    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:33.179787 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:33.179721    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:43.179860 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:43.179808    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:53.179294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.179225    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:47:53.448226 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.448145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-56fc49c9b5-6z45b_79323635-0922-47c8-8966-12c7f847dd52/main/0.log"
Jul 31 15:47:53.448639 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.448614    2557 generic.go:356] "Generic (PLEG): container finished" podID="79323635-0922-47c8-8966-12c7f847dd52" containerID="8f0942c7421a9c634f9c03e0b583df1c250b9163ecab249351adfb2bcc5fbc5a" exitCode=137
Jul 31 15:47:53.448749 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.448686    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerDied","Data":"8f0942c7421a9c634f9c03e0b583df1c250b9163ecab249351adfb2bcc5fbc5a"}
Jul 31 15:47:53.492203 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.492173    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-56fc49c9b5-6z45b_79323635-0922-47c8-8966-12c7f847dd52/main/0.log"
Jul 31 15:47:53.494800 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.494772    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:47:53.599026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.598984    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599033    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599342 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599070    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599342 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599089    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599342 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599167    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599342 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599186    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599342 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599268    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599624 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599357    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnj69\" (UniqueName: \"kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69\") pod \"79323635-0922-47c8-8966-12c7f847dd52\" (UID: \"79323635-0922-47c8-8966-12c7f847dd52\") "
Jul 31 15:47:53.599685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599614    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:47:53.599685 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599646    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:47:53.599817 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599780    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home" (OuterVolumeSpecName: "home") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:47:53.599928 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.599901    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache" (OuterVolumeSpecName: "model-cache") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:47:53.600145 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.600126    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.600206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.600151    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.600206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.600165    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.600206 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.600178    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79323635-0922-47c8-8966-12c7f847dd52-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.601726 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.601698    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm" (OuterVolumeSpecName: "dshm") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:47:53.601969 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.601951    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69" (OuterVolumeSpecName: "kube-api-access-jnj69") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "kube-api-access-jnj69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:47:53.602124 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.602107    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:47:53.621637 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.621586    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79323635-0922-47c8-8966-12c7f847dd52" (UID: "79323635-0922-47c8-8966-12c7f847dd52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:47:53.701524 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.701432    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.701524 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.701463    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79323635-0922-47c8-8966-12c7f847dd52-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.701524 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.701475    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnj69\" (UniqueName: \"kubernetes.io/projected/79323635-0922-47c8-8966-12c7f847dd52-kube-api-access-jnj69\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:53.701524 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:53.701488    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79323635-0922-47c8-8966-12c7f847dd52-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:47:54.455976 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.455947    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-56fc49c9b5-6z45b_79323635-0922-47c8-8966-12c7f847dd52/main/0.log"
Jul 31 15:47:54.456514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.456436    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b" event={"ID":"79323635-0922-47c8-8966-12c7f847dd52","Type":"ContainerDied","Data":"dfde94f3bf7d134050ad559a4d800f32a060a4135522d7dbcd05e23884bf3c51"}
Jul 31 15:47:54.456514 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.456494    2557 scope.go:117] "RemoveContainer" containerID="8f0942c7421a9c634f9c03e0b583df1c250b9163ecab249351adfb2bcc5fbc5a"
Jul 31 15:47:54.456644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.456442    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"
Jul 31 15:47:54.466140 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.466113    2557 scope.go:117] "RemoveContainer" containerID="64a73f1767c97e0e0f08d75f5f29b024bb1d0c68587d8a310eb8403745b6210d"
Jul 31 15:47:54.477533 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.477498    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:47:54.480834 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:54.480798    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-56fc49c9b5-6z45b"]
Jul 31 15:47:56.249776 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:56.249659    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79323635-0922-47c8-8966-12c7f847dd52" path="/var/lib/kubelet/pods/79323635-0922-47c8-8966-12c7f847dd52/volumes"
Jul 31 15:47:58.494369 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:58.494327    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:47:58.507851 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:47:58.507797    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:48:03.179358 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:03.179303    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:48:13.179126 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:13.179076    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8001/health\": dial tcp 10.132.0.63:8001: connect: connection refused"
Jul 31 15:48:23.190260 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:23.190202    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:48:23.203659 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:23.203627    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:48:42.818593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:42.818554    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:48:57.842103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:57.842041    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main" containerID="cri-o://ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1" gracePeriod=45
Jul 31 15:48:58.459442 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.459401    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="llm-d-routing-sidecar" containerID="cri-o://b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f" gracePeriod=45
Jul 31 15:48:58.709218 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.709191    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:48:58.720088 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720063    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720098    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720148    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720170    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720188    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720294 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720231    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720319    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720361    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrqqh\" (UniqueName: \"kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh\") pod \"727037e0-11aa-4d70-890a-6a208351af1b\" (UID: \"727037e0-11aa-4d70-890a-6a208351af1b\") "
Jul 31 15:48:58.720593 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720534    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.720756 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.720680    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.721258 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.721212    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:48:58.721395 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.721284    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home" (OuterVolumeSpecName: "home") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.723051 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.723026    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm" (OuterVolumeSpecName: "dshm") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.723188 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.723127    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:48:58.723188 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.723136    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh" (OuterVolumeSpecName: "kube-api-access-qrqqh") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "kube-api-access-qrqqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:48:58.738332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.738296    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.753867 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.753826    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "727037e0-11aa-4d70-890a-6a208351af1b" (UID: "727037e0-11aa-4d70-890a-6a208351af1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:48:58.791664 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791621    2557 generic.go:356] "Generic (PLEG): container finished" podID="727037e0-11aa-4d70-890a-6a208351af1b" containerID="ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1" exitCode=0
Jul 31 15:48:58.791664 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791650    2557 generic.go:356] "Generic (PLEG): container finished" podID="727037e0-11aa-4d70-890a-6a208351af1b" containerID="b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f" exitCode=0
Jul 31 15:48:58.791914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791677    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerDied","Data":"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"}
Jul 31 15:48:58.791914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791719    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerDied","Data":"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"}
Jul 31 15:48:58.791914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791735    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n" event={"ID":"727037e0-11aa-4d70-890a-6a208351af1b","Type":"ContainerDied","Data":"7a287b504be6a8eb001b7a58c0615665a31477db5711e646ea4a185127858366"}
Jul 31 15:48:58.791914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791678    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"
Jul 31 15:48:58.791914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.791755    2557 scope.go:117] "RemoveContainer" containerID="ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"
Jul 31 15:48:58.807125 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.807091    2557 scope.go:117] "RemoveContainer" containerID="0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"
Jul 31 15:48:58.815730 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.815703    2557 scope.go:117] "RemoveContainer" containerID="b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"
Jul 31 15:48:58.818551 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.818525    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:48:58.821580 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821558    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/727037e0-11aa-4d70-890a-6a208351af1b-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821580 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821585    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrqqh\" (UniqueName: \"kubernetes.io/projected/727037e0-11aa-4d70-890a-6a208351af1b-kube-api-access-qrqqh\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821602    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821617    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727037e0-11aa-4d70-890a-6a208351af1b-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821657    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821669    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.821766 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.821681    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727037e0-11aa-4d70-890a-6a208351af1b-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:48:58.825367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.825313    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-9c5689f55-hq66n"]
Jul 31 15:48:58.827215 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.827195    2557 scope.go:117] "RemoveContainer" containerID="ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"
Jul 31 15:48:58.827567 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:48:58.827543    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1\": container with ID starting with ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1 not found: ID does not exist" containerID="ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"
Jul 31 15:48:58.827638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.827577    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"} err="failed to get container status \"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1\": rpc error: code = NotFound desc = could not find container \"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1\": container with ID starting with ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1 not found: ID does not exist"
Jul 31 15:48:58.827638 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.827597    2557 scope.go:117] "RemoveContainer" containerID="0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"
Jul 31 15:48:58.827870 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:48:58.827850    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b\": container with ID starting with 0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b not found: ID does not exist" containerID="0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"
Jul 31 15:48:58.827914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.827881    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"} err="failed to get container status \"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b\": rpc error: code = NotFound desc = could not find container \"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b\": container with ID starting with 0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b not found: ID does not exist"
Jul 31 15:48:58.827914 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.827899    2557 scope.go:117] "RemoveContainer" containerID="b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"
Jul 31 15:48:58.828117 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:48:58.828101    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f\": container with ID starting with b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f not found: ID does not exist" containerID="b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"
Jul 31 15:48:58.828161 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828121    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"} err="failed to get container status \"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f\": rpc error: code = NotFound desc = could not find container \"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f\": container with ID starting with b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f not found: ID does not exist"
Jul 31 15:48:58.828161 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828134    2557 scope.go:117] "RemoveContainer" containerID="ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"
Jul 31 15:48:58.828345 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828325    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1"} err="failed to get container status \"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1\": rpc error: code = NotFound desc = could not find container \"ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1\": container with ID starting with ea4ef22ab3a1ee526a79189ba08dadb32ac1e734cabec4e07cb6fdea215d23e1 not found: ID does not exist"
Jul 31 15:48:58.828397 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828347    2557 scope.go:117] "RemoveContainer" containerID="0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"
Jul 31 15:48:58.828574 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828555    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b"} err="failed to get container status \"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b\": rpc error: code = NotFound desc = could not find container \"0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b\": container with ID starting with 0d1eb6b550259071d0b535804da2164c641cf738729eadcf4bbb87217becba0b not found: ID does not exist"
Jul 31 15:48:58.828618 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828575    2557 scope.go:117] "RemoveContainer" containerID="b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"
Jul 31 15:48:58.828790 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:48:58.828774    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f"} err="failed to get container status \"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f\": rpc error: code = NotFound desc = could not find container \"b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f\": container with ID starting with b198faa57c700a6b3ccca2b8ecfca970786bdc74e11a8b1df77fdde0a8e3642f not found: ID does not exist"
Jul 31 15:49:00.248133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:00.248096    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727037e0-11aa-4d70-890a-6a208351af1b" path="/var/lib/kubelet/pods/727037e0-11aa-4d70-890a-6a208351af1b/volumes"
Jul 31 15:49:11.327442 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327407    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:49:11.327956 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327899    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="storage-initializer"
Jul 31 15:49:11.327956 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327918    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="storage-initializer"
Jul 31 15:49:11.327956 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327953    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="storage-initializer"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327962    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="storage-initializer"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327972    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="main"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327980    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="main"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.327993    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="llm-d-routing-sidecar"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328002    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="llm-d-routing-sidecar"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328014    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328024    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328110    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="llm-d-routing-sidecar"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328122    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="79323635-0922-47c8-8966-12c7f847dd52" containerName="main"
Jul 31 15:49:11.328133 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.328132    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="727037e0-11aa-4d70-890a-6a208351af1b" containerName="main"
Jul 31 15:49:11.333175 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.333151    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.336103 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336040    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:49:11.336273 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336050    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-tntkm\""
Jul 31 15:49:11.336354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336305    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 31 15:49:11.336354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336327    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 31 15:49:11.336354 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336311    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 31 15:49:11.336771 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.336755    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 31 15:49:11.342044 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.342022    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:49:11.432705 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.432657    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433033 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433015    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433174 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433158    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433340 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433321    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433447    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9rg\" (UniqueName: \"kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433605 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433591    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433727 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433713    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.433848 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.433834    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534599 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534553    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534807 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534633    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534807 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534666    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534807 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534695    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534838    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9rg\" (UniqueName: \"kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534888    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534924    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.534989 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.534953    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.535187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.535057    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.535187 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.535100    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.535357 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.535330    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.535541 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.535374    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.535631 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.535553    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.537142 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.537119    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.537271 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.537219    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.545393 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.545366    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9rg\" (UniqueName: \"kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.644792 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.644687    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:11.791553 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.791516    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:49:11.795218 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:49:11.795192    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2641be3_9294_4fb7_acac_42eb06c975cf.slice/crio-72b1ad38103a3e28b6fea0fdafd07091d0d963e3aa0f4a349ce2e926528fcbc6 WatchSource:0}: Error finding container 72b1ad38103a3e28b6fea0fdafd07091d0d963e3aa0f4a349ce2e926528fcbc6: Status 404 returned error can't find the container with id 72b1ad38103a3e28b6fea0fdafd07091d0d963e3aa0f4a349ce2e926528fcbc6
Jul 31 15:49:11.857213 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:11.857182    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerStarted","Data":"72b1ad38103a3e28b6fea0fdafd07091d0d963e3aa0f4a349ce2e926528fcbc6"}
Jul 31 15:49:12.863339 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:12.863297    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerStarted","Data":"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"}
Jul 31 15:49:14.874557 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:14.874523    2557 generic.go:356] "Generic (PLEG): container finished" podID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerID="114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37" exitCode=0
Jul 31 15:49:14.874978 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:14.874598    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerDied","Data":"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"}
Jul 31 15:49:15.882295 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:15.882259    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerStarted","Data":"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"}
Jul 31 15:49:15.906555 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:15.906500    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podStartSLOduration=2.6197913120000003 podStartE2EDuration="4.906480903s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.286689591s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:49:15.904462691 +0000 UTC m=+3678.062714180" watchObservedRunningTime="2026-07-31 15:49:15.906480903 +0000 UTC m=+3678.064732393"
Jul 31 15:49:16.645047 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:16.645011    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:21.645568 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:21.645466    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:49:21.647255 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:21.647208    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:49:31.645808 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:31.645744    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:49:41.645309 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:41.645265    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:49:51.645827 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:49:51.645787    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:01.645818 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:01.645769    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:11.646198 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:11.646143    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:21.645450 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:21.645380    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:31.645513 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:31.645433    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:41.645366 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:41.645294    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:50:51.645935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:50:51.645835    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:51:01.645204 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:01.645157    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:51:11.645327 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:11.645279    2557 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:51:21.655249 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:21.655211    2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:51:21.663631 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:21.663601    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:51:26.478118 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:26.478079    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:51:41.501188 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:41.501123    2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" containerID="cri-o://c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763" gracePeriod=45
Jul 31 15:51:41.656155 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:41.656114    2557 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 31 15:51:42.253706 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.253679    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:51:42.372111 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372016    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372111 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372086    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372130    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372167    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372199    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372226    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372309    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372389 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372336    2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb9rg\" (UniqueName: \"kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg\") pod \"a2641be3-9294-4fb7-acac-42eb06c975cf\" (UID: \"a2641be3-9294-4fb7-acac-42eb06c975cf\") "
Jul 31 15:51:42.372709 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372444    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:51:42.372777 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372752    2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-model-cache\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.372777 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372757    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:51:42.372881 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.372831    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home" (OuterVolumeSpecName: "home") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:51:42.374545 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.374520    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm" (OuterVolumeSpecName: "dshm") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:51:42.374820 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.374787    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:51:42.374935 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.374894    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg" (OuterVolumeSpecName: "kube-api-access-mb9rg") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "kube-api-access-mb9rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:51:42.386397 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.386356    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:51:42.397746 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.397702    2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2641be3-9294-4fb7-acac-42eb06c975cf" (UID: "a2641be3-9294-4fb7-acac-42eb06c975cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:51:42.473758 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473714    2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-home\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.473758 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473749    2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2641be3-9294-4fb7-acac-42eb06c975cf-tls-certs\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.473758 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473763    2557 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-kserve-provision-location\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.474026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473777    2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-tmp-dir\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.474026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473788    2557 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2641be3-9294-4fb7-acac-42eb06c975cf-cabundle-cert\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.474026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473799    2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9rg\" (UniqueName: \"kubernetes.io/projected/a2641be3-9294-4fb7-acac-42eb06c975cf-kube-api-access-mb9rg\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.474026 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.473810    2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2641be3-9294-4fb7-acac-42eb06c975cf-dshm\") on node \"ip-10-0-137-96.ec2.internal\" DevicePath \"\""
Jul 31 15:51:42.615782 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.615750    2557 generic.go:356] "Generic (PLEG): container finished" podID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerID="c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763" exitCode=0
Jul 31 15:51:42.616385 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.615803    2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"
Jul 31 15:51:42.616385 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.615833    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerDied","Data":"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"}
Jul 31 15:51:42.616385 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.615890    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4" event={"ID":"a2641be3-9294-4fb7-acac-42eb06c975cf","Type":"ContainerDied","Data":"72b1ad38103a3e28b6fea0fdafd07091d0d963e3aa0f4a349ce2e926528fcbc6"}
Jul 31 15:51:42.616385 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.615914    2557 scope.go:117] "RemoveContainer" containerID="c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"
Jul 31 15:51:42.627689 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.627671    2557 scope.go:117] "RemoveContainer" containerID="114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"
Jul 31 15:51:42.642339 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.642309    2557 scope.go:117] "RemoveContainer" containerID="c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"
Jul 31 15:51:42.642737 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:51:42.642716    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763\": container with ID starting with c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763 not found: ID does not exist" containerID="c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"
Jul 31 15:51:42.642836 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.642745    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763"} err="failed to get container status \"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763\": rpc error: code = NotFound desc = could not find container \"c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763\": container with ID starting with c87973df4f95b9f4b9ab85f7971922717bfde61f1281cdfe8331b81a3d5e8763 not found: ID does not exist"
Jul 31 15:51:42.642836 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.642764    2557 scope.go:117] "RemoveContainer" containerID="114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"
Jul 31 15:51:42.643024 ip-10-0-137-96 kubenswrapper[2557]: E0731 15:51:42.643000    2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37\": container with ID starting with 114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37 not found: ID does not exist" containerID="114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"
Jul 31 15:51:42.643072 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.643033    2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37"} err="failed to get container status \"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37\": rpc error: code = NotFound desc = could not find container \"114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37\": container with ID starting with 114de9e0b19328db8b6a4ec59f0d48c204d4d39d9061158756a7c22d88d2ad37 not found: ID does not exist"
Jul 31 15:51:42.643757 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.643740    2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:51:42.647730 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:42.647704    2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5d9588c688ll4q4"]
Jul 31 15:51:44.248232 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:51:44.248193    2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" path="/var/lib/kubelet/pods/a2641be3-9294-4fb7-acac-42eb06c975cf/volumes"
Jul 31 15:52:15.004894 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:15.004864    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-lnx9l_d1a46c41-8870-4367-8be0-09a75a1a29df/manager/0.log"
Jul 31 15:52:15.014054 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:15.014025    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-8q6sk_bc93181f-bdf0-49eb-bf34-621bb5309908/limitador/0.log"
Jul 31 15:52:23.977863 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:23.977781    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jhwrc_368ae851-9ae7-4d58-8715-aa95f856f689/global-pull-secret-syncer/0.log"
Jul 31 15:52:24.151179 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:24.151145    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wmscn_34476e65-7f26-44fc-824f-0824aae840c6/konnectivity-agent/0.log"
Jul 31 15:52:24.222877 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:24.222837    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-96.ec2.internal_cf43c7428d90fe73c7fe683bbce5474e/haproxy/0.log"
Jul 31 15:52:29.428579 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:29.428540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-lnx9l_d1a46c41-8870-4367-8be0-09a75a1a29df/manager/0.log"
Jul 31 15:52:29.468124 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:29.468040    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-8q6sk_bc93181f-bdf0-49eb-bf34-621bb5309908/limitador/0.log"
Jul 31 15:52:30.858262 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.858216    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/alertmanager/0.log"
Jul 31 15:52:30.881486 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.881459    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/config-reloader/0.log"
Jul 31 15:52:30.909523 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.909497    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/kube-rbac-proxy-web/0.log"
Jul 31 15:52:30.935956 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.935929    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/kube-rbac-proxy/0.log"
Jul 31 15:52:30.964154 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.964121    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/kube-rbac-proxy-metric/0.log"
Jul 31 15:52:30.993570 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:30.993540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/prom-label-proxy/0.log"
Jul 31 15:52:31.024615 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.024588    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9975b93a-f021-4c64-a616-c070a21c4886/init-config-reloader/0.log"
Jul 31 15:52:31.080223 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.080131    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-vbzzx_5a8fb483-e8a9-45e6-aabb-7ec50eb47049/cluster-monitoring-operator/0.log"
Jul 31 15:52:31.259841 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.259815    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-85m9f_d2379fd1-ed3c-4629-b028-ab394a084506/monitoring-plugin/0.log"
Jul 31 15:52:31.437959 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.437927    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lt8hw_d6c3e29d-3c0b-47cc-802e-722d3697e9b8/node-exporter/0.log"
Jul 31 15:52:31.463074 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.463051    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lt8hw_d6c3e29d-3c0b-47cc-802e-722d3697e9b8/kube-rbac-proxy/0.log"
Jul 31 15:52:31.505114 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.505086    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lt8hw_d6c3e29d-3c0b-47cc-802e-722d3697e9b8/init-textfile/0.log"
Jul 31 15:52:31.779951 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.779916    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/prometheus/0.log"
Jul 31 15:52:31.801946 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.801919    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/config-reloader/0.log"
Jul 31 15:52:31.826709 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.826673    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/thanos-sidecar/0.log"
Jul 31 15:52:31.854049 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.854022    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/kube-rbac-proxy-web/0.log"
Jul 31 15:52:31.882657 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.882627    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/kube-rbac-proxy/0.log"
Jul 31 15:52:31.909732 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.909707    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/kube-rbac-proxy-thanos/0.log"
Jul 31 15:52:31.936439 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.936409    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c1c424a8-c7fe-48dd-8688-f0fb80d41479/init-config-reloader/0.log"
Jul 31 15:52:31.969670 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.969621    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-x9wc7_8d598830-f534-4769-9200-7d04baee3a2d/prometheus-operator/0.log"
Jul 31 15:52:31.991484 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:31.991455    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-758d7cb874-x9wc7_8d598830-f534-4769-9200-7d04baee3a2d/kube-rbac-proxy/0.log"
Jul 31 15:52:32.023961 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:32.023932    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-dq9h6_a2b394d2-edfb-4b26-8f18-e296e9f6dbbd/prometheus-operator-admission-webhook/0.log"
Jul 31 15:52:32.056195 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:32.056093    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94cb45949-qxdwh_dbe1f860-18ee-443f-91c5-076bfcc88504/telemeter-client/0.log"
Jul 31 15:52:32.092422 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:32.092393    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94cb45949-qxdwh_dbe1f860-18ee-443f-91c5-076bfcc88504/reload/0.log"
Jul 31 15:52:32.116077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:32.116020    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-94cb45949-qxdwh_dbe1f860-18ee-443f-91c5-076bfcc88504/kube-rbac-proxy/0.log"
Jul 31 15:52:33.326474 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.326433    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-h5rrh_b80a5438-cc6d-4b63-ae98-7f10c04cdf4c/networking-console-plugin/0.log"
Jul 31 15:52:33.818773 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.818739    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/2.log"
Jul 31 15:52:33.823634 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.823594    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-7ltxn_b6504bf4-ba53-48c5-aa00-91d40893f55b/console-operator/3.log"
Jul 31 15:52:33.890584 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890549    2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"]
Jul 31 15:52:33.890910 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890898    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="storage-initializer"
Jul 31 15:52:33.890957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890912    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="storage-initializer"
Jul 31 15:52:33.890957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890921    2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main"
Jul 31 15:52:33.890957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890926    2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main"
Jul 31 15:52:33.891058 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.890998    2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2641be3-9294-4fb7-acac-42eb06c975cf" containerName="main"
Jul 31 15:52:33.894334 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.894312    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:33.897769 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.897742    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"openshift-service-ca.crt\""
Jul 31 15:52:33.897903 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.897832    2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bs68h\"/\"kube-root-ca.crt\""
Jul 31 15:52:33.898883 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.898858    2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bs68h\"/\"default-dockercfg-85wq2\""
Jul 31 15:52:33.906207 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:33.906178    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"]
Jul 31 15:52:34.001175 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.001133    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-lib-modules\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.001408 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.001201    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-sys\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.001408 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.001225    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-proc\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.001408 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.001364    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdgnm\" (UniqueName: \"kubernetes.io/projected/47afc242-4cf2-44e6-8532-6ca7e6f82758-kube-api-access-vdgnm\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.001408 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.001402    2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-podres\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102590    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-podres\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102698 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102650    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-lib-modules\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102705    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-sys\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102723    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-proc\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102757    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-podres\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102769    2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdgnm\" (UniqueName: \"kubernetes.io/projected/47afc242-4cf2-44e6-8532-6ca7e6f82758-kube-api-access-vdgnm\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102811    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-sys\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102822    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-proc\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.102970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.102852    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47afc242-4cf2-44e6-8532-6ca7e6f82758-lib-modules\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.110796 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.110760    2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdgnm\" (UniqueName: \"kubernetes.io/projected/47afc242-4cf2-44e6-8532-6ca7e6f82758-kube-api-access-vdgnm\") pod \"perf-node-gather-daemonset-b4hf5\" (UID: \"47afc242-4cf2-44e6-8532-6ca7e6f82758\") " pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.204970 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.204921    2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.339519 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.339484    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-wn6s5_33af7aec-f40c-4baa-bed3-0b02839031e2/download-server/0.log"
Jul 31 15:52:34.355329 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.355306    2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"]
Jul 31 15:52:34.358150 ip-10-0-137-96 kubenswrapper[2557]: W0731 15:52:34.358113    2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod47afc242_4cf2_44e6_8532_6ca7e6f82758.slice/crio-32e3866c794e43e0eeeb6414fe23036d1d49f5fc26466de8886489c8293692c1 WatchSource:0}: Error finding container 32e3866c794e43e0eeeb6414fe23036d1d49f5fc26466de8886489c8293692c1: Status 404 returned error can't find the container with id 32e3866c794e43e0eeeb6414fe23036d1d49f5fc26466de8886489c8293692c1
Jul 31 15:52:34.859006 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.858966    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5" event={"ID":"47afc242-4cf2-44e6-8532-6ca7e6f82758","Type":"ContainerStarted","Data":"46ff20110e0e4eae76f2ef72d8d317944c29266bab110ec5d8bdde4cffd1ded7"}
Jul 31 15:52:34.859201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.859023    2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5" event={"ID":"47afc242-4cf2-44e6-8532-6ca7e6f82758","Type":"ContainerStarted","Data":"32e3866c794e43e0eeeb6414fe23036d1d49f5fc26466de8886489c8293692c1"}
Jul 31 15:52:34.859201 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.859047    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:34.876957 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:34.876897    2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5" podStartSLOduration=1.87687902 podStartE2EDuration="1.87687902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:52:34.875955925 +0000 UTC m=+3877.034207415" watchObservedRunningTime="2026-07-31 15:52:34.87687902 +0000 UTC m=+3877.035130530"
Jul 31 15:52:35.926901 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:35.926861    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkpw6_f90ef8e2-afdd-4440-8c3f-5d268609ba6e/dns/0.log"
Jul 31 15:52:35.949332 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:35.949302    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lkpw6_f90ef8e2-afdd-4440-8c3f-5d268609ba6e/kube-rbac-proxy/0.log"
Jul 31 15:52:36.033889 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:36.033846    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g8ptv_88b37453-42ef-47fc-97fe-ad3c86bee43f/dns-node-resolver/0.log"
Jul 31 15:52:36.644034 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:36.643997    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2k6ch_3b43d8aa-a300-479e-abab-9ef368b0df29/node-ca/0.log"
Jul 31 15:52:38.554710 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:38.554672    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tvb95_58d74f63-ce55-4673-9752-584c1d65522f/serve-healthcheck-canary/0.log"
Jul 31 15:52:39.028857 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:39.028817    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wdlx9_f0a42421-6f22-42af-99a2-15f971f5c897/insights-operator/0.log"
Jul 31 15:52:39.029708 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:39.029684    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-wdlx9_f0a42421-6f22-42af-99a2-15f971f5c897/insights-operator/1.log"
Jul 31 15:52:39.193077 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:39.193045    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2756_4ce77561-80c9-4f58-aba1-ebbe60e62527/kube-rbac-proxy/0.log"
Jul 31 15:52:39.215513 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:39.215477    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2756_4ce77561-80c9-4f58-aba1-ebbe60e62527/exporter/0.log"
Jul 31 15:52:39.237772 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:39.237736    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n2756_4ce77561-80c9-4f58-aba1-ebbe60e62527/extractor/0.log"
Jul 31 15:52:40.875506 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:40.875464    2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bs68h/perf-node-gather-daemonset-b4hf5"
Jul 31 15:52:42.280584 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:42.280546    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56854c75c8-hmj5l_f7c84927-c6ea-46d7-8390-746b208cf3c8/manager/0.log"
Jul 31 15:52:50.680954 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:50.680911    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2m6nz_2790d032-2163-47fc-8d82-18241f606df8/kube-multus/0.log"
Jul 31 15:52:51.040021 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.039942    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:52:51.064048 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.064017    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/egress-router-binary-copy/0.log"
Jul 31 15:52:51.090367 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.090337    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/cni-plugins/0.log"
Jul 31 15:52:51.114567 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.114540    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/bond-cni-plugin/0.log"
Jul 31 15:52:51.136531 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.136501    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/routeoverride-cni/0.log"
Jul 31 15:52:51.160644 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.160611    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/whereabouts-cni-bincopy/0.log"
Jul 31 15:52:51.182339 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.182303    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-znz82_b1b80295-cf3a-4204-9e2f-2703eef3aa50/whereabouts-cni/0.log"
Jul 31 15:52:51.276922 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.276885    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jf5r_f1ad65b5-6b4d-4d71-8ca8-b9326a58b567/network-metrics-daemon/0.log"
Jul 31 15:52:51.296300 ip-10-0-137-96 kubenswrapper[2557]: I0731 15:52:51.296180    2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6jf5r_f1ad65b5-6b4d-4d71-8ca8-b9326a58b567/kube-rbac-proxy/0.log"