Jul 29 10:26:56.672368 ip-10-0-133-86 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 29 10:26:56.672377 ip-10-0-133-86 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 29 10:26:56.672384 ip-10-0-133-86 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 29 10:26:56.672660 ip-10-0-133-86 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 29 10:27:06.737155 ip-10-0-133-86 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 29 10:27:06.737177 ip-10-0-133-86 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e4879885c84c4306896f7a63c94f06b0 --
Jul 29 10:28:43.694574 ip-10-0-133-86 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 10:28:44.125711 ip-10-0-133-86 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:44.125711 ip-10-0-133-86 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 10:28:44.125711 ip-10-0-133-86 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:44.125711 ip-10-0-133-86 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 10:28:44.125711 ip-10-0-133-86 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 10:28:44.126737 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.126653    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 10:28:44.131671 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131651    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.131671 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131671    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131675    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131688    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131691    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131694    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131697    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131699    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131702    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131704    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131707    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131710    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131712    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131715    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131717    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131720    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131722    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131725    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131728    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131730    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131733    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.131747 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131744    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131747    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131750    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131753    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131762    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131765    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131768    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131770    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131773    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131775    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131778    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131780    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131783    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131786    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131789    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131791    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131793    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131796    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131798    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131801    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.132234 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131803    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131805    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131808    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131811    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131814    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131816    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131819    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131821    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131823    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131827    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131829    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131832    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131834    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131843    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131846    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131850    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131858    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131861    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131864    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.132831 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131866    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131869    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131872    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131875    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131879    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131884    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131886    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131889    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131891    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131894    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131896    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131899    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131902    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131904    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131906    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131909    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131912    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131915    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131917    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131920    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.133697 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131922    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131925    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131927    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131930    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131933    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131935    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131939    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131948    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131951    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131954    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131957    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131959    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131962    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131964    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131969    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131972    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131975    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131978    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.131980    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.134526 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133640    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133655    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133659    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133664    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133668    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133672    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133676    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133680    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133684    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133689    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133693    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133916    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133939    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133947    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133952    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133957    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133961    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133966    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133970    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.135374 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133975    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133979    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133984    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133991    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.133997    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134002    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134007    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134012    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134017    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134021    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134026    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134030    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134035    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134040    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134044    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134048    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134053    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134063    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134068    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134073    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.136194 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134077    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134081    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134085    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134089    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134093    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134099    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134104    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134108    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134113    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134120    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134125    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134129    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134133    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134138    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134142    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134146    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134151    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134155    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134159    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134164    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.137100 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134168    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134173    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134177    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134181    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134185    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134190    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134194    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134198    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134202    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134206    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134211    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134236    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134241    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134245    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134249    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134254    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134259    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134264    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134271    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.137660 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134276    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134280    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134285    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134288    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134293    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134297    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134302    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134306    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134310    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134315    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134319    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134323    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134327    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134331    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134335    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134340    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134344    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134348    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134352    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134357    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.138206 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.134364    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135050    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135060    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135065    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135071    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135075    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135080    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135085    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135092    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135097    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135103    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135108    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135112    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135117    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135121    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135125    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135129    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135135    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135139    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135143    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.138805 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135148    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135152    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135156    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135160    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135165    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135169    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135173    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135177    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135181    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135185    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135189    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135196    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135202    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135206    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135211    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135235    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135240    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135245    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135249    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135253    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.139450 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135257    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135261    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135267    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135271    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135276    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135280    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135284    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135289    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135294    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135300    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135308    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135312    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135316    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135320    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135324    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135328    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135333    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135337    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135341    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.140050 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135345    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135349    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135353    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135358    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135362    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135366    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135370    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135375    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135379    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135383    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135387    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135390    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135394    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135398    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135404    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135408    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135413    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135417    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135421    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135426    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.140548 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135430    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135435    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135439    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135443    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135447    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135452    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135456    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135461    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135465    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135469    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135473    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135478    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135482    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135486    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135490    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135494    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135498    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135501    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135506    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.141114 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135511    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.135515    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135632    2574 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135645    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135655    2574 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135662    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135669    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135674    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135685    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135693    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135698    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135703    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135709    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135714    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135719    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135723    2574 flags.go:64] FLAG: --cgroup-root=""
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135728    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135733    2574 flags.go:64] FLAG: --client-ca-file=""
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135739    2574 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135744    2574 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135753    2574 flags.go:64] FLAG: --cluster-domain=""
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135758    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135763    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135768    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 10:28:44.141652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135773    2574 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135785    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135790    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135795    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135800    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135805    2574 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135810    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135815    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135821    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135826    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135833    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135838    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135842    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135847    2574 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135852    2574 flags.go:64] FLAG: --enable-server="true"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135857    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135866    2574 flags.go:64] FLAG: --event-burst="100"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135870    2574 flags.go:64] FLAG: --event-qps="50"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135875    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135880    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135885    2574 flags.go:64] FLAG: --eviction-hard=""
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135892    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135896    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135901    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135906    2574 flags.go:64] FLAG: --eviction-soft=""
Jul 29 10:28:44.142269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135911    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135916    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135922    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135926    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135931    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135936    2574 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135940    2574 flags.go:64] FLAG: --feature-gates=""
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135947    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135952    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135957    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135962    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135967    2574 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135972    2574 flags.go:64] FLAG: --help="false"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135976    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135982    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135987    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135992    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.135997    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136003    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136008    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136013    2574 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136017    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136021    2574 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 10:28:44.142858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136028    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136033    2574 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136037    2574 flags.go:64] FLAG: --kube-reserved=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136042    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136047    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136052    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136056    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136061    2574 flags.go:64] FLAG: --lock-file=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136066    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136070    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136075    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136085    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136091    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136096    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136101    2574 flags.go:64] FLAG: --logging-format="text"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136105    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136111    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136115    2574 flags.go:64] FLAG: --manifest-url=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136120    2574 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136133    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136138    2574 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136144    2574 flags.go:64] FLAG: --max-pods="110"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136149    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136156    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136161    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 10:28:44.143479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136165    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136170    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136175    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136179    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136192    2574 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136197    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136201    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136207    2574 flags.go:64] FLAG: --pod-cidr=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136212    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136238    2574 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136243    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136248    2574 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136252    2574 flags.go:64] FLAG: --port="10250"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136257    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136262    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d696f8ff61a1cc9"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136268    2574 flags.go:64] FLAG: --qos-reserved=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136272    2574 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136278    2574 flags.go:64] FLAG: --register-node="true"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136282    2574 flags.go:64] FLAG: --register-with-taints=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136288    2574 flags.go:64] FLAG: --registry-burst="10"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136294    2574 flags.go:64] FLAG: --registry-qps="5"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136299    2574 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136304    2574 flags.go:64] FLAG: --reserved-memory=""
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136310    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136316    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 10:28:44.144167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136320    2574 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136325    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136330    2574 flags.go:64] FLAG: --runonce="false"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136334    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136339    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136347    2574 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136351    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136356    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136361    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136366    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136371    2574 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136376    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136380    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136385    2574 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136390    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136396    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136401    2574 flags.go:64] FLAG: --system-cgroups=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136406    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136415    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136420    2574 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136424    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136432    2574 flags.go:64] FLAG: --tls-min-version=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136436    2574 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136441    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136446    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 10:28:44.144840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136451    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136456    2574 flags.go:64] FLAG: --v="2"
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136462    2574 flags.go:64] FLAG: --version="false"
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136470    2574 flags.go:64] FLAG: --vmodule=""
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136477    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.136482    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136674    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136681    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136687    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136691    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136696    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136700    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136707    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136711    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136715    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136719    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136723    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136727    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136732    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136736    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136740    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.145521 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136744    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136751    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136755    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136762    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136768    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136772    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136777    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136781    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136785    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136791    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136795    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136800    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136804    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136808    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136812    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136817    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136821    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136826    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136830    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136834    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.146047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136838    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136843    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136847    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136854    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136858    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136862    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136866    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136871    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136875    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136879    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136883    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136888    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136892    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136897    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136901    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136906    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136910    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136917    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136922    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.146571 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136927    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136931    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136936    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136941    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136946    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136950    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136955    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136960    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136964    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136969    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136974    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136980    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136986    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136990    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136994    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.136999    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137004    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137008    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137013    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.147047 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137017    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137021    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137025    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137030    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137034    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137038    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137042    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137048    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137052    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137056    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137060    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137064    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137069    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137073    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137076    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137081    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137085    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137089    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137093    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137096    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.147550 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137101    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137105    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137108    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137113    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137116    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.137120    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.137746    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.145875    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.145897    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.145917    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.145923    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.145986    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.145991    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.145995    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.145998    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.148042 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146000    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146003    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146006    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146009    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146013    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146017    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146020    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146023    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146026    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146028    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146031    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146034    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146036    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146039    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146041    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146044    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146046    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146049    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146052    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146054    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.148583 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146057    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146060    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146063    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146065    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146068    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146072    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146075    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146079    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146082    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146085    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146088    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146091    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146094    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146096    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146099    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146102    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146104    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146107    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146109    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.149064 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146112    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146114    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146117    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146120    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146122    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146125    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146128    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146130    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146133    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146135    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146137    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146140    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146142    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146145    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146148    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146150    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146153    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146156    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146159    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.149574 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146169    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146173    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146175    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146178    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146181    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146183    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146186    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146188    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146191    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146194    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146196    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146199    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146201    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146204    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146207    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146209    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146212    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146235    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146238    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.150046 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146241    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146243    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146246    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146249    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146252    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146254    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146257    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146259    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146263    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146267    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146271    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146274    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146277    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146281    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146290    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146293    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146296    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.150532 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146299    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.146304    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146431    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146436    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146440    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146442    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146445    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146448    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146450    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146453    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146455    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146458    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146461    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146463    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146466    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 10:28:44.150959 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146468    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146471    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146474    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146476    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146478    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146481    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146483    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146486    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146488    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146491    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146493    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146496    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146499    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146502    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146505    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146508    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146511    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146513    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146516    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 10:28:44.151353 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146518    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146521    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146523    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146526    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146528    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146532    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146535    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146539    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146542    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146545    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146547    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146550    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146552    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146554    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146557    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146559    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146562    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146564    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146567    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146587    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 10:28:44.151814 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146591    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146594    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146597    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146599    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146603    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146606    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146608    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146611    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146613    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146622    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146624    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146627    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146630    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146632    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146635    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146638    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146640    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146643    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146646    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146648    2574 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 10:28:44.152322 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146650    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146653    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146655    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146658    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146660    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146663    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146665    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146668    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146670    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146672    2574 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146675    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146677    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146680    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146682    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146686    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146689    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146692    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146695    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146698    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 10:28:44.152798 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146701    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146705    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146708    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146711    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146714    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146716    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146719    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:44.146721    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.146726    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.146870    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.149195    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.150671    2574 server.go:1019] "Starting client certificate rotation"
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.150780    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:44.153278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.150845    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 10:28:44.153608 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.153044    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:44.154301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.154199    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 10:28:44.164383 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.164318    2574 log.go:25] "Validated CRI v1 runtime API"
Jul 29 10:28:44.169328 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.169306    2574 log.go:25] "Validated CRI v1 image API"
Jul 29 10:28:44.170199 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.170182    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 10:28:44.173025 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.173004    2574 fs.go:135] Filesystem UUIDs: map[2f86ed25-48d7-44a3-8612-64f70009c040:/dev/nvme0n1p4 53e0408f-af61-4ecc-b882-f0b04738ba59:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 29 10:28:44.173090 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.173024    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 10:28:44.179123 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.178993    2574 manager.go:216] Machine: {Timestamp:2026-07-29 10:28:44.176727397 +0000 UTC m=+0.357109267 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097780 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e717690bc66a088ec522b31ff8389 SystemUUID:ec2e7176-90bc-66a0-88ec-522b31ff8389 BootID:e4879885-c84c-4306-896f-7a63c94f06b0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6b:7d:e7:80:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6b:7d:e7:80:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:2f:b4:dc:8d:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 10:28:44.179123 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.179110    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 10:28:44.179293 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.179209    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 10:28:44.180124 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.180104    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:44.183012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.182986    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 10:28:44.183171 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.183015    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-86.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 10:28:44.183840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.183830    2574 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 10:28:44.183876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.183844    2574 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 10:28:44.183876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.183851    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:44.184542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.184531    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 10:28:44.185336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.185325    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 10:28:44.188039 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188023    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:44.188184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188172    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 10:28:44.188241 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188205    2574 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 10:28:44.188314 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188300    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 10:28:44.188361 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188326    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 10:28:44.188361 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188340    2574 kubelet.go:398] "Adding apiserver pod source"
Jul 29 10:28:44.188361 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188350    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 10:28:44.188765 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188751    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:44.188836 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.188775    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 10:28:44.191102 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.191080    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 10:28:44.193166 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.193144    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 10:28:44.193296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.193181    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 10:28:44.194821 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194805    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194826    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194836    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194843    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194852    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194866    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194873    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194879    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194886    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194891    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 10:28:44.194894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194900    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 10:28:44.195195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.194907    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 10:28:44.196508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.196485    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 10:28:44.196508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.196503    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 10:28:44.199272 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.199248    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-86.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 10:28:44.199628 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.199602    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 10:28:44.199665 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.199631    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 10:28:44.200339 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.200324    2574 server.go:1268] "Started kubelet"
Jul 29 10:28:44.200471 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.200393    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 10:28:44.200533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.200443    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 10:28:44.200608 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.200541    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 10:28:44.201495 ip-10-0-133-86 systemd[1]: Started Kubernetes Kubelet.
Jul 29 10:28:44.204690 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.204652    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 10:28:44.206071 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.206042    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 10:28:44.206760 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.206743    2574 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 10:28:44.211590 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.210606    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-86.ec2.internal.18c6bbebe907c091  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.ec2.internal,UID:ip-10-0-133-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.ec2.internal,},FirstTimestamp:2026-07-29 10:28:44.200288401 +0000 UTC m=+0.380670273,LastTimestamp:2026-07-29 10:28:44.200288401 +0000 UTC m=+0.380670273,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.ec2.internal,}"
Jul 29 10:28:44.214319 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.214293    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:44.214414 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.214303    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 10:28:44.214473 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.214433    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.214951 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.214921    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 10:28:44.215072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.214917    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 10:28:44.215161 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215089    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 10:28:44.215287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215198    2574 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 10:28:44.215287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215284    2574 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 10:28:44.215776 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215748    2574 factory.go:153] Registering CRI-O factory
Jul 29 10:28:44.215835 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215788    2574 factory.go:223] Registration of the crio container factory successfully
Jul 29 10:28:44.215865 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215845    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 10:28:44.215865 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215856    2574 factory.go:55] Registering systemd factory
Jul 29 10:28:44.215865 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215864    2574 factory.go:223] Registration of the systemd container factory successfully
Jul 29 10:28:44.215968 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215884    2574 factory.go:103] Registering Raw factory
Jul 29 10:28:44.215968 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.215900    2574 manager.go:1196] Started watching for new ooms in manager
Jul 29 10:28:44.216627 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.216614    2574 manager.go:349] Starting recovery of all containers
Jul 29 10:28:44.219860 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.219833    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-86.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 10:28:44.219860 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.219840    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 10:28:44.226645 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.226519    2574 manager.go:354] Recovery completed
Jul 29 10:28:44.229630 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.229611    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdfm7"
Jul 29 10:28:44.230919 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.230905    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.232764 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.232751    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.232826 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.232777    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.232826 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.232787    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.234121 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234107    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 10:28:44.234121 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234119    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 10:28:44.234236 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234135    2574 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 10:28:44.234236 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234113    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.234565 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.234499    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-86.ec2.internal.18c6bbebeaf74ce1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-86.ec2.internal,UID:ip-10-0-133-86.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-86.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-86.ec2.internal,},FirstTimestamp:2026-07-29 10:28:44.232764641 +0000 UTC m=+0.413146511,LastTimestamp:2026-07-29 10:28:44.232764641 +0000 UTC m=+0.413146511,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-86.ec2.internal,}"
Jul 29 10:28:44.234937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234922    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.235003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234948    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.235003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.234958    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.236834 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.236822    2574 policy_none.go:49] "None policy: Start"
Jul 29 10:28:44.236888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.236839    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 10:28:44.236888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.236849    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 10:28:44.237164 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.237150    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bdfm7"
Jul 29 10:28:44.238860 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.238849    2574 policy_none.go:47] "Start"
Jul 29 10:28:44.275823 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.275795    2574 manager.go:337] "Starting Device Plugin manager"
Jul 29 10:28:44.275952 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.275839    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 10:28:44.275952 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.275853    2574 server.go:87] "Starting device plugin registration server"
Jul 29 10:28:44.276142 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.276128    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 10:28:44.276203 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.276143    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 10:28:44.276303 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.276283    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 10:28:44.276380 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.276368    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 10:28:44.276419 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.276381    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 10:28:44.276876 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.276858    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 10:28:44.276942 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.276901    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.288224 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.288200    2574 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2610 into the system container \"/system.slice\": <nil>"
Jul 29 10:28:44.334141 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.334097    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 10:28:44.336291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.335512    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 10:28:44.336291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.335549    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 10:28:44.336291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.335575    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 10:28:44.336291 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.335610    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 10:28:44.338412 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.338387    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:44.376995 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.376899    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.377913 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.377894    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.378004 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.377927    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.378004 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.377939    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.378004 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.377963    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.388754 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.388728    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.388754 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.388753    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-86.ec2.internal\": node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.414795 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.414765    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.436000 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.435969    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"]
Jul 29 10:28:44.436081 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.436049    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.437707 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.437688    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.437800 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.437720    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.437800 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.437730    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.439999 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.439986    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.440109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.440095    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.440163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.440155    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.440784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.440769    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.440853 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.440796    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.440853 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.440806    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.441391 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.441367    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.441391 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.441399    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.441526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.441408    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.443606 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.443586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.443688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.443613    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 10:28:44.444399 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.444375    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 10:28:44.444492 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.444408    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 10:28:44.444492 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.444424    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeHasSufficientPID"
Jul 29 10:28:44.475126 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.475097    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-86.ec2.internal\" not found" node="ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.479882 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.479866    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-86.ec2.internal\" not found" node="ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.515277 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.515246    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.516417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.516390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.516417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.516419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.516567 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.516437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d1203cd46ad847a51fd730c0dc9ddcf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.ec2.internal\" (UID: \"0d1203cd46ad847a51fd730c0dc9ddcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.615832 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.615788    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.616940 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.616913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.617001 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.616960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.617001 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.616964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.617078 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.616998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6e2c422f24a43001d363c82a2d67531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal\" (UID: \"f6e2c422f24a43001d363c82a2d67531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.617078 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.617012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d1203cd46ad847a51fd730c0dc9ddcf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.ec2.internal\" (UID: \"0d1203cd46ad847a51fd730c0dc9ddcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.617078 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.617047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d1203cd46ad847a51fd730c0dc9ddcf-config\") pod \"kube-apiserver-proxy-ip-10-0-133-86.ec2.internal\" (UID: \"0d1203cd46ad847a51fd730c0dc9ddcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.716508 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.716439    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.777717 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.777697    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.782882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:44.782759    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:44.816951 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.816908    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:44.917448 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:44.917397    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:45.017949 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.017887    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-86.ec2.internal\" not found"
Jul 29 10:28:45.061777 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.061752    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.082140 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.082112    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.115601 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.115565    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:45.129477 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.129453    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:45.131622 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.131607    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal"
Jul 29 10:28:45.140549 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.140531    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 10:28:45.149695 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.149670    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 10:28:45.149834 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.149815    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:45.149890 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.149837    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:45.149890 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.149845    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 10:28:45.189046 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.189009    2574 apiserver.go:52] "Watching apiserver"
Jul 29 10:28:45.205345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.205317    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 10:28:45.205688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.205666    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-9ncwb","openshift-multus/network-metrics-daemon-5fm96","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal","openshift-multus/multus-additional-cni-plugins-lk64z","openshift-network-diagnostics/network-check-target-mrh9c","openshift-network-operator/iptables-alerter-bqvvn","openshift-ovn-kubernetes/ovnkube-node-jkg9m","kube-system/global-pull-secret-syncer-dsndc","kube-system/konnectivity-agent-dsdrb","kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xkgp2","openshift-dns/node-resolver-gl58w","openshift-image-registry/node-ca-6nvz2"]
Jul 29 10:28:45.208263 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.208244    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.210334 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.210312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.210473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.210457    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 10:28:45.210536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.210485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z4xxv\""
Jul 29 10:28:45.210596 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.210537    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 10:28:45.212195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.212154    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 10:28:45.212771 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.212752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.212894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.212873    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.213184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.212796    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nqlh9\""
Jul 29 10:28:45.213335 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.213246    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.214560 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.214542    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 10:28:45.214689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.214570    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 10:28:45.214791 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.214776    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.215142 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.215118    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.215256 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.215205    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j6p8b\""
Jul 29 10:28:45.215351 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.215331    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.217201 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217186    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 10:28:45.217308 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217297    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.217416 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217401    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5p886\""
Jul 29 10:28:45.217480 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.217603 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217590    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.217685 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217668    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 10:28:45.217984 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.217968    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 10:28:45.218150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.218137    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 10:28:45.219066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/642a5976-8b42-47d0-a96b-7885f7385cf6-ovn-node-metrics-cert\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3043fc9-285c-4a12-88c9-3566d826375a-agent-certs\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.219120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knm52\" (UniqueName: \"kubernetes.io/projected/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-kube-api-access-knm52\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.219120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6c58d40-a2da-4b45-aa50-091fe6d34cca-serviceca\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219122    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-systemd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-etc-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-config\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219261    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-iptables-alerter-script\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-slash\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-ovn\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-bin\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25hgf\" (UniqueName: \"kubernetes.io/projected/642a5976-8b42-47d0-a96b-7885f7385cf6-kube-api-access-25hgf\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2818e4ae-44ec-460b-a866-86b544e12a30-hosts-file\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219431    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-systemd-units\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219449    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-script-lib\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219473    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfn79\" (UniqueName: \"kubernetes.io/projected/a6c58d40-a2da-4b45-aa50-091fe6d34cca-kube-api-access-kfn79\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-env-overrides\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219549    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7xnhw\""
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3043fc9-285c-4a12-88c9-3566d826375a-konnectivity-ca\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-var-lib-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219639    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-log-socket\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-netd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.219784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219734    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.219800    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89z6b\" (UniqueName: \"kubernetes.io/projected/2818e4ae-44ec-460b-a866-86b544e12a30-kube-api-access-89z6b\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-host-slash\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-kubelet\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-netns\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.219982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2818e4ae-44ec-460b-a866-86b544e12a30-tmp-dir\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.220048    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c58d40-a2da-4b45-aa50-091fe6d34cca-host\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.220408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.220096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-node-log\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.221864 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.221849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.221918 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.221903    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:45.224113 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.224093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.226304 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.226288    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.226389 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.226348    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:45.227339 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227323    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 10:28:45.227339 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227330    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.227461 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227343    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 10:28:45.227461 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227361    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.227539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227492    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 10:28:45.227592 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.227578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dqcsf\""
Jul 29 10:28:45.228677 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.228660    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.229148 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.229133    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 10:28:45.230418 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.230396    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 10:28:45.230418 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.230407    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mvcrt\""
Jul 29 10:28:45.230932 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.230921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.232539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.232524    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.232844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.232829    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rkjxd\""
Jul 29 10:28:45.232844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.232841    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.233142 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.233125    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.235051 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.235036    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 10:28:45.235123 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.235060    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 10:28:45.235183 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.235137    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjgpl\""
Jul 29 10:28:45.235183 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.235165    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 10:28:45.241713 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.241668    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 10:23:44 +0000 UTC" deadline="2028-04-23 12:38:33.72946868 +0000 UTC"
Jul 29 10:28:45.241713 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.241711    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15218h9m48.487759768s"
Jul 29 10:28:45.253457 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.253440    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8g4r"
Jul 29 10:28:45.261338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.261317    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f8g4r"
Jul 29 10:28:45.268065 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.267938    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:45.315892 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.315875    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 10:28:45.320711 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-os-release\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.320771 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3043fc9-285c-4a12-88c9-3566d826375a-konnectivity-ca\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.320771 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-log-socket\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.320839 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-netd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.320879 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320831    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-log-socket\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.320879 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-netd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.320879 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89z6b\" (UniqueName: \"kubernetes.io/projected/2818e4ae-44ec-460b-a866-86b544e12a30-kube-api-access-89z6b\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.321005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-multus\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.321005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-lib-modules\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.321005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-sys-fs\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.321005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.320983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-kubelet\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321257 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-netns\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321257 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-netns\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321257 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pbn\" (UniqueName: \"kubernetes.io/projected/2df1e671-4dab-4eed-b983-326d3fc53b9c-kube-api-access-v4pbn\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.321257 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-kubelet\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-etc-kubernetes\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c58d40-a2da-4b45-aa50-091fe6d34cca-host\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-node-log\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c3043fc9-285c-4a12-88c9-3566d826375a-konnectivity-ca\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjt2\" (UniqueName: \"kubernetes.io/projected/5442ddcf-c235-46d4-a3bb-faace9f9f281-kube-api-access-cwjt2\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6c58d40-a2da-4b45-aa50-091fe6d34cca-host\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321403    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6c58d40-a2da-4b45-aa50-091fe6d34cca-serviceca\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-node-log\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-systemd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321525    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-systemd\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321562    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-os-release\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321595    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-socket-dir-parent\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321629    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-kubernetes\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-sys\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-etc-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-config\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321746    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-etc-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-dbus\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6c58d40-a2da-4b45-aa50-091fe6d34cca-serviceca\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.321846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321815    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321890    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-kubelet\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-tuned\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25hgf\" (UniqueName: \"kubernetes.io/projected/642a5976-8b42-47d0-a96b-7885f7385cf6-kube-api-access-25hgf\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.321993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2818e4ae-44ec-460b-a866-86b544e12a30-hosts-file\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-cnibin\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-netns\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-run\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-tmp\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2818e4ae-44ec-460b-a866-86b544e12a30-hosts-file\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9m97\" (UniqueName: \"kubernetes.io/projected/46fdf764-65bc-4ba2-a8c7-114fd18dac81-kube-api-access-z9m97\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-systemd-units\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-systemd-units\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-script-lib\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-system-cni-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.322539 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-binary-copy\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-conf-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-daemon-config\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn79\" (UniqueName: \"kubernetes.io/projected/a6c58d40-a2da-4b45-aa50-091fe6d34cca-kube-api-access-kfn79\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322381    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cni-binary-copy\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr75d\" (UniqueName: \"kubernetes.io/projected/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-kube-api-access-mr75d\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-host\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-env-overrides\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-var-lib-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-var-lib-openvswitch\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-hostroot\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-modprobe-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-host-slash\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-script-lib\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2818e4ae-44ec-460b-a866-86b544e12a30-tmp-dir\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-bin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322786    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-run-ovn-kubernetes\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-host-slash\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-systemd\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-device-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwlf\" (UniqueName: \"kubernetes.io/projected/b579a599-b611-43a6-b50d-2e9b5a936c32-kube-api-access-lqwlf\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322856    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/642a5976-8b42-47d0-a96b-7885f7385cf6-ovn-node-metrics-cert\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322861    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-env-overrides\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322902    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/642a5976-8b42-47d0-a96b-7885f7385cf6-ovnkube-config\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cnibin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-socket-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.322981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3043fc9-285c-4a12-88c9-3566d826375a-agent-certs\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2818e4ae-44ec-460b-a866-86b544e12a30-tmp-dir\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knm52\" (UniqueName: \"kubernetes.io/projected/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-kube-api-access-knm52\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-kubelet-config\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.323884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysconfig\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323118    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-registration-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323141    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-iptables-alerter-script\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-slash\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-slash\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-ovn\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-bin\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323431    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-run-ovn\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-system-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323484    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/642a5976-8b42-47d0-a96b-7885f7385cf6-host-cni-bin\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-k8s-cni-cncf-io\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-multus-certs\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-iptables-alerter-script\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.324843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-var-lib-kubelet\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.325669 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:45.325669 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.325669 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.323697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-conf\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.326307 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.326291    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/642a5976-8b42-47d0-a96b-7885f7385cf6-ovn-node-metrics-cert\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.326408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.326393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c3043fc9-285c-4a12-88c9-3566d826375a-agent-certs\") pod \"konnectivity-agent-dsdrb\" (UID: \"c3043fc9-285c-4a12-88c9-3566d826375a\") " pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.328879 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.328857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89z6b\" (UniqueName: \"kubernetes.io/projected/2818e4ae-44ec-460b-a866-86b544e12a30-kube-api-access-89z6b\") pod \"node-resolver-gl58w\" (UID: \"2818e4ae-44ec-460b-a866-86b544e12a30\") " pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.329746 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.329729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25hgf\" (UniqueName: \"kubernetes.io/projected/642a5976-8b42-47d0-a96b-7885f7385cf6-kube-api-access-25hgf\") pod \"ovnkube-node-jkg9m\" (UID: \"642a5976-8b42-47d0-a96b-7885f7385cf6\") " pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.330002 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.329964    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn79\" (UniqueName: \"kubernetes.io/projected/a6c58d40-a2da-4b45-aa50-091fe6d34cca-kube-api-access-kfn79\") pod \"node-ca-6nvz2\" (UID: \"a6c58d40-a2da-4b45-aa50-091fe6d34cca\") " pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.330127 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.330113    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knm52\" (UniqueName: \"kubernetes.io/projected/e1c66da3-c1dc-4e85-9ae2-79ae5c1db565-kube-api-access-knm52\") pod \"iptables-alerter-bqvvn\" (UID: \"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565\") " pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.424347 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-registration-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.424347 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-system-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424409    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-registration-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-k8s-cni-cncf-io\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-k8s-cni-cncf-io\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-system-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424526    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-multus-certs\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424564    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-var-lib-kubelet\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-conf\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-var-lib-kubelet\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424619    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-multus-certs\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-os-release\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.424704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-multus\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-lib-modules\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-sys-fs\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pbn\" (UniqueName: \"kubernetes.io/projected/2df1e671-4dab-4eed-b983-326d3fc53b9c-kube-api-access-v4pbn\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-etc-kubernetes\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424755    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysctl-conf\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-os-release\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424771    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-multus\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjt2\" (UniqueName: \"kubernetes.io/projected/5442ddcf-c235-46d4-a3bb-faace9f9f281-kube-api-access-cwjt2\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424812    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-os-release\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424831    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-etc-kubernetes\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424830    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-sys-fs\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424838    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-lib-modules\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-os-release\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-socket-dir-parent\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.425536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-kubernetes\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-sys\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424956    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-socket-dir-parent\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-dbus\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.424981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-kubernetes\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425006    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-sys\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-kubelet\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425100    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-dbus\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-tuned\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-cnibin\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-cni-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.425152    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-netns\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425187    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-run\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.426313 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-tmp\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.425246    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:45.925211164 +0000 UTC m=+2.105593022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9m97\" (UniqueName: \"kubernetes.io/projected/46fdf764-65bc-4ba2-a8c7-114fd18dac81-kube-api-access-z9m97\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-system-cni-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425330    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-binary-copy\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-conf-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-daemon-config\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425398    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-run-netns\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425414    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425439    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cni-binary-copy\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425444    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-cnibin\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr75d\" (UniqueName: \"kubernetes.io/projected/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-kube-api-access-mr75d\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-run\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425501    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-kubelet\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-host\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-hostroot\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-modprobe-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425602    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-bin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425649    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-systemd\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-device-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwlf\" (UniqueName: \"kubernetes.io/projected/b579a599-b611-43a6-b50d-2e9b5a936c32-kube-api-access-lqwlf\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5442ddcf-c235-46d4-a3bb-faace9f9f281-system-cni-dir\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cnibin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-socket-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-kubelet-config\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.425821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysconfig\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-daemon-config\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-multus-conf-dir\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426074    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-sysconfig\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426123    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-host\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.427926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-hostroot\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426207    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cnibin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5442ddcf-c235-46d4-a3bb-faace9f9f281-cni-binary-copy\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-socket-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426353    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-host-var-lib-cni-bin\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-kubelet-config\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426405    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b579a599-b611-43a6-b50d-2e9b5a936c32-device-dir\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.426439    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426442    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-systemd\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426486    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-modprobe-d\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.426499    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:45.926482116 +0000 UTC m=+2.106863990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.426842    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-cni-binary-copy\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.427531    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-etc-tuned\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.428748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.427559    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46fdf764-65bc-4ba2-a8c7-114fd18dac81-tmp\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.432406 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.432383    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:45.432406 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.432407    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:45.432575 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.432420    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:45.432575 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.432485    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:45.932467026 +0000 UTC m=+2.112848884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:45.434406 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.434357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9m97\" (UniqueName: \"kubernetes.io/projected/46fdf764-65bc-4ba2-a8c7-114fd18dac81-kube-api-access-z9m97\") pod \"tuned-xkgp2\" (UID: \"46fdf764-65bc-4ba2-a8c7-114fd18dac81\") " pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.434909 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.434879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pbn\" (UniqueName: \"kubernetes.io/projected/2df1e671-4dab-4eed-b983-326d3fc53b9c-kube-api-access-v4pbn\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.434909 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.434879    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjt2\" (UniqueName: \"kubernetes.io/projected/5442ddcf-c235-46d4-a3bb-faace9f9f281-kube-api-access-cwjt2\") pod \"multus-additional-cni-plugins-lk64z\" (UID: \"5442ddcf-c235-46d4-a3bb-faace9f9f281\") " pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.435080 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.435012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwlf\" (UniqueName: \"kubernetes.io/projected/b579a599-b611-43a6-b50d-2e9b5a936c32-kube-api-access-lqwlf\") pod \"aws-ebs-csi-driver-node-8hl6k\" (UID: \"b579a599-b611-43a6-b50d-2e9b5a936c32\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.435529 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.435511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr75d\" (UniqueName: \"kubernetes.io/projected/085098cb-f1e3-4b6a-ac1d-ef7528e75bcb-kube-api-access-mr75d\") pod \"multus-9ncwb\" (UID: \"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb\") " pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.503306 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.503275    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e2c422f24a43001d363c82a2d67531.slice/crio-9017fa924a3e45317e27417b01fd9e721db207b1c147d44557ff8aa09173d214 WatchSource:0}: Error finding container 9017fa924a3e45317e27417b01fd9e721db207b1c147d44557ff8aa09173d214: Status 404 returned error can't find the container with id 9017fa924a3e45317e27417b01fd9e721db207b1c147d44557ff8aa09173d214
Jul 29 10:28:45.508438 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.508421    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:28:45.529034 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.528949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:28:45.538765 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.538740    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3043fc9_285c_4a12_88c9_3566d826375a.slice/crio-2fac5039483d95d476600df16e931c24d4a53f904c6a42d38563fe2c9044447c WatchSource:0}: Error finding container 2fac5039483d95d476600df16e931c24d4a53f904c6a42d38563fe2c9044447c: Status 404 returned error can't find the container with id 2fac5039483d95d476600df16e931c24d4a53f904c6a42d38563fe2c9044447c
Jul 29 10:28:45.559756 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.559728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqvvn"
Jul 29 10:28:45.568451 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.568422    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c66da3_c1dc_4e85_9ae2_79ae5c1db565.slice/crio-c68190169854c34b2e269f6c10a41fa438ab7d0d52247dcbc4d3aa265318360e WatchSource:0}: Error finding container c68190169854c34b2e269f6c10a41fa438ab7d0d52247dcbc4d3aa265318360e: Status 404 returned error can't find the container with id c68190169854c34b2e269f6c10a41fa438ab7d0d52247dcbc4d3aa265318360e
Jul 29 10:28:45.578807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.578787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6nvz2"
Jul 29 10:28:45.582638 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.582616    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:28:45.587438 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.587409    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c58d40_a2da_4b45_aa50_091fe6d34cca.slice/crio-4c7166b1c8b951e276ebc14aa5336e69081a4cbaca6d738dbad86af20f7b33e5 WatchSource:0}: Error finding container 4c7166b1c8b951e276ebc14aa5336e69081a4cbaca6d738dbad86af20f7b33e5: Status 404 returned error can't find the container with id 4c7166b1c8b951e276ebc14aa5336e69081a4cbaca6d738dbad86af20f7b33e5
Jul 29 10:28:45.591168 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.591147    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642a5976_8b42_47d0_a96b_7885f7385cf6.slice/crio-25a5c0835a43d4447851a5e28c193b3e4a055522a98e923bb0b936294af0fadf WatchSource:0}: Error finding container 25a5c0835a43d4447851a5e28c193b3e4a055522a98e923bb0b936294af0fadf: Status 404 returned error can't find the container with id 25a5c0835a43d4447851a5e28c193b3e4a055522a98e923bb0b936294af0fadf
Jul 29 10:28:45.610252 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.610205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gl58w"
Jul 29 10:28:45.615076 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.615054    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lk64z"
Jul 29 10:28:45.617260 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.617237    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ncwb"
Jul 29 10:28:45.621434 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.621401    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2818e4ae_44ec_460b_a866_86b544e12a30.slice/crio-5a9880574af26173449b6b88931c8f9a6e80efa9ceb351753af0320c5aad2581 WatchSource:0}: Error finding container 5a9880574af26173449b6b88931c8f9a6e80efa9ceb351753af0320c5aad2581: Status 404 returned error can't find the container with id 5a9880574af26173449b6b88931c8f9a6e80efa9ceb351753af0320c5aad2581
Jul 29 10:28:45.626244 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.626156    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5442ddcf_c235_46d4_a3bb_faace9f9f281.slice/crio-70a80bec3375eabd1eb355606f5b0511e6f4eb8cc1fc922387c6f5e98c0ab831 WatchSource:0}: Error finding container 70a80bec3375eabd1eb355606f5b0511e6f4eb8cc1fc922387c6f5e98c0ab831: Status 404 returned error can't find the container with id 70a80bec3375eabd1eb355606f5b0511e6f4eb8cc1fc922387c6f5e98c0ab831
Jul 29 10:28:45.628441 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.628413    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085098cb_f1e3_4b6a_ac1d_ef7528e75bcb.slice/crio-ab304065b16e2600edada4d8685abed7091718fb717746fa5a162d1f31e6b8cb WatchSource:0}: Error finding container ab304065b16e2600edada4d8685abed7091718fb717746fa5a162d1f31e6b8cb: Status 404 returned error can't find the container with id ab304065b16e2600edada4d8685abed7091718fb717746fa5a162d1f31e6b8cb
Jul 29 10:28:45.650356 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.650325    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xkgp2"
Jul 29 10:28:45.658461 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.658429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k"
Jul 29 10:28:45.659727 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.659703    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fdf764_65bc_4ba2_a8c7_114fd18dac81.slice/crio-6dbe52d7fc15768ef7ccd60caed41804c8f08b59c2d5c7ed7c7d28dcc8eaab21 WatchSource:0}: Error finding container 6dbe52d7fc15768ef7ccd60caed41804c8f08b59c2d5c7ed7c7d28dcc8eaab21: Status 404 returned error can't find the container with id 6dbe52d7fc15768ef7ccd60caed41804c8f08b59c2d5c7ed7c7d28dcc8eaab21
Jul 29 10:28:45.666775 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.666750    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb579a599_b611_43a6_b50d_2e9b5a936c32.slice/crio-11df8da59f1155c2e95b37f6e0ffd44c14e684731ca72e00e6d4b87214004602 WatchSource:0}: Error finding container 11df8da59f1155c2e95b37f6e0ffd44c14e684731ca72e00e6d4b87214004602: Status 404 returned error can't find the container with id 11df8da59f1155c2e95b37f6e0ffd44c14e684731ca72e00e6d4b87214004602
Jul 29 10:28:45.929077 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.929037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:45.929301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:45.929127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:45.929301 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.929242    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.929388 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.929347    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:46.929324765 +0000 UTC m=+3.109706628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:45.929706 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.929264    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.929773 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:45.929764    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:46.929731679 +0000 UTC m=+3.110113539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:45.935742 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:28:45.935702    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1203cd46ad847a51fd730c0dc9ddcf.slice/crio-7d5aa5af393d1d6adcd47e4c1b9a01db32b9c005094b89c3b7bd588cb4011151 WatchSource:0}: Error finding container 7d5aa5af393d1d6adcd47e4c1b9a01db32b9c005094b89c3b7bd588cb4011151: Status 404 returned error can't find the container with id 7d5aa5af393d1d6adcd47e4c1b9a01db32b9c005094b89c3b7bd588cb4011151
Jul 29 10:28:46.031216 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.030343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:46.031216 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.030579    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:46.031216 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.030603    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:46.031216 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.030618    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:46.031216 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.030698    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:47.030676789 +0000 UTC m=+3.211058673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:46.033097 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.033073    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:46.264251 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.264121    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:45 +0000 UTC" deadline="2028-03-12 02:58:33.867768305 +0000 UTC"
Jul 29 10:28:46.264251 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.264188    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14200h29m47.60358472s"
Jul 29 10:28:46.367615 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.367518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6nvz2" event={"ID":"a6c58d40-a2da-4b45-aa50-091fe6d34cca","Type":"ContainerStarted","Data":"4c7166b1c8b951e276ebc14aa5336e69081a4cbaca6d738dbad86af20f7b33e5"}
Jul 29 10:28:46.376525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.376468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsdrb" event={"ID":"c3043fc9-285c-4a12-88c9-3566d826375a","Type":"ContainerStarted","Data":"2fac5039483d95d476600df16e931c24d4a53f904c6a42d38563fe2c9044447c"}
Jul 29 10:28:46.381572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.381532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal" event={"ID":"f6e2c422f24a43001d363c82a2d67531","Type":"ContainerStarted","Data":"9017fa924a3e45317e27417b01fd9e721db207b1c147d44557ff8aa09173d214"}
Jul 29 10:28:46.385137 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.384829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal" event={"ID":"0d1203cd46ad847a51fd730c0dc9ddcf","Type":"ContainerStarted","Data":"7d5aa5af393d1d6adcd47e4c1b9a01db32b9c005094b89c3b7bd588cb4011151"}
Jul 29 10:28:46.398546 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.398506    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k" event={"ID":"b579a599-b611-43a6-b50d-2e9b5a936c32","Type":"ContainerStarted","Data":"11df8da59f1155c2e95b37f6e0ffd44c14e684731ca72e00e6d4b87214004602"}
Jul 29 10:28:46.402352 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.402316    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gl58w" event={"ID":"2818e4ae-44ec-460b-a866-86b544e12a30","Type":"ContainerStarted","Data":"5a9880574af26173449b6b88931c8f9a6e80efa9ceb351753af0320c5aad2581"}
Jul 29 10:28:46.406585 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.406547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"25a5c0835a43d4447851a5e28c193b3e4a055522a98e923bb0b936294af0fadf"}
Jul 29 10:28:46.412243 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.412201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqvvn" event={"ID":"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565","Type":"ContainerStarted","Data":"c68190169854c34b2e269f6c10a41fa438ab7d0d52247dcbc4d3aa265318360e"}
Jul 29 10:28:46.418846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.418805    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkgp2" event={"ID":"46fdf764-65bc-4ba2-a8c7-114fd18dac81","Type":"ContainerStarted","Data":"6dbe52d7fc15768ef7ccd60caed41804c8f08b59c2d5c7ed7c7d28dcc8eaab21"}
Jul 29 10:28:46.436565 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.436519    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ncwb" event={"ID":"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb","Type":"ContainerStarted","Data":"ab304065b16e2600edada4d8685abed7091718fb717746fa5a162d1f31e6b8cb"}
Jul 29 10:28:46.446728 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.446612    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerStarted","Data":"70a80bec3375eabd1eb355606f5b0511e6f4eb8cc1fc922387c6f5e98c0ab831"}
Jul 29 10:28:46.811077 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.811042    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:46.939182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.939138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:46.939389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:46.939250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:46.939474 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.939455    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:46.939550 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.939529    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:48.93950602 +0000 UTC m=+5.119887905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:46.940079 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.940034    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:46.940162 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:46.940091    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:48.940073581 +0000 UTC m=+5.120455461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:47.040151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.040109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:47.040392 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.040370    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:47.040471 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.040398    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:47.040471 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.040413    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:47.040582 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.040479    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:49.040458723 +0000 UTC m=+5.220840599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:47.264486 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.264352    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 10:23:45 +0000 UTC" deadline="2028-02-13 09:22:40.344922177 +0000 UTC"
Jul 29 10:28:47.264486 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.264426    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h53m53.080501631s"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.336566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.336735    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.337170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.337265    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.337355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:47.337610 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:47.337433    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:47.377438 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:47.377402    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 10:28:48.957405 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:48.957321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:48.957868 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:48.957416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:48.957868 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:48.957590    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.957868 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:48.957658    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:28:52.957639748 +0000 UTC m=+9.138021622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:48.958162 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:48.958096    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:48.958162 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:48.958159    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:28:52.958141831 +0000 UTC m=+9.138523713 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:49.058508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:49.058468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:49.058710 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.058670    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:49.058710 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.058693    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:49.058710 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.058709    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.058882 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.058800    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:28:53.058779034 +0000 UTC m=+9.239160913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:49.336631    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.336792    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:49.337212    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.337334    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:49.337385    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:49.337642 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:49.337458    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:51.336799 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:51.336758    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:51.337268 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:51.336905    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:51.337422 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:51.337390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:51.337422 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:51.337408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:51.337628 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:51.337524    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:51.337628 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:51.337607    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:52.994822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:52.995062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:52.995067    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:52.995135    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:00.995113448 +0000 UTC m=+17.175495306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:52.995178    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:52.995279 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:52.995247    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:00.995233147 +0000 UTC m=+17.175615019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:28:53.096632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:53.095994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:53.096632 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.096157    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:28:53.096632 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.096178    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:28:53.096632 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.096192    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:53.096632 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.096273    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:01.0962502 +0000 UTC m=+17.276632083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:28:53.336393 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:53.336313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:53.336550 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.336487    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:53.336550 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:53.336315    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:53.336660 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.336598    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:53.336660 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:53.336313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:53.336737 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:53.336675    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:55.335993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:55.336139    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:55.336236    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:55.336304    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:55.336350    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:55.336447 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:55.336416    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:57.336473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:57.336440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:57.336936 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:57.336579    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:57.337006 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:57.336944    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:57.337006 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:57.336977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:57.337146 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:57.337115    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:57.337252 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:57.337208    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:28:59.336011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:59.335972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:28:59.336469 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:59.336078    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:28:59.336469 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:59.335990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:28:59.336469 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:28:59.336127    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:28:59.336469 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:59.336248    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:28:59.336469 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:28:59.336341    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:01.056370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.056333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:01.056899 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.056409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:01.056899 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.056496    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.056899 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.056541    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:01.056899 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.056570    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.056548085 +0000 UTC m=+33.236929960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:01.056899 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.056595    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.05657742 +0000 UTC m=+33.236959299 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:01.157682 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.157647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:01.157875 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.157826    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:01.157875 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.157847    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:01.157875 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.157856    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.158015 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.157913    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:17.157896831 +0000 UTC m=+33.338278691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:01.336011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.335924    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:01.336011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.335964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:01.336011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:01.335964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:01.336426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.336072    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:01.336426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.336149    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:01.336426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:01.336210    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:03.336098 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:03.336059    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:03.336631 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:03.336119    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:03.336631 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:03.336070    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:03.336631 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:03.336235    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:03.336631 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:03.336336    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:03.336852 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:03.336419    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:05.336873 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:05.336829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:05.336873 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:05.336873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:05.337426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:05.336843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:05.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:05.336973    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:05.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:05.337060    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:05.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:05.337143    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:07.336814 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.336630    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:07.337426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.336635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:07.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:07.336906    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:07.337426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.336675    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:07.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:07.336963    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:07.337426 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:07.337031    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:07.503841 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.503815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xkgp2" event={"ID":"46fdf764-65bc-4ba2-a8c7-114fd18dac81","Type":"ContainerStarted","Data":"bed2f96b9f09907d52f06108b50831d75ab70461ab07602f2393741c58dc5256"}
Jul 29 10:29:07.505130 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.505102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ncwb" event={"ID":"085098cb-f1e3-4b6a-ac1d-ef7528e75bcb","Type":"ContainerStarted","Data":"958ee3aa39c69db76112e368eae3e7a6c8b8afab2aa778a5cf8de8e395da916d"}
Jul 29 10:29:07.506551 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.506527    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="4428ad09a13cb4e5297f1f837621f8d63788b09159da7b40537d29c5ee88f08c" exitCode=0
Jul 29 10:29:07.506612 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.506597    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"4428ad09a13cb4e5297f1f837621f8d63788b09159da7b40537d29c5ee88f08c"}
Jul 29 10:29:07.507953 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.507934    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6nvz2" event={"ID":"a6c58d40-a2da-4b45-aa50-091fe6d34cca","Type":"ContainerStarted","Data":"a997413530434d6f08ad2bc1272c5fae545792f9acf15826ca04011d50548dbd"}
Jul 29 10:29:07.509393 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.509368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dsdrb" event={"ID":"c3043fc9-285c-4a12-88c9-3566d826375a","Type":"ContainerStarted","Data":"b2960471fbe087b0fc227a61840ec7a7c67287e81e4543394bc0cb8862fb20d8"}
Jul 29 10:29:07.510938 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.510754    2574 generic.go:356] "Generic (PLEG): container finished" podID="f6e2c422f24a43001d363c82a2d67531" containerID="4ee978cbbc814a7d891d5471752114e75200f1b9557aa9e72e05dadfb71c574f" exitCode=0
Jul 29 10:29:07.511036 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.510818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal" event={"ID":"f6e2c422f24a43001d363c82a2d67531","Type":"ContainerDied","Data":"4ee978cbbc814a7d891d5471752114e75200f1b9557aa9e72e05dadfb71c574f"}
Jul 29 10:29:07.512547 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.512518    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal" event={"ID":"0d1203cd46ad847a51fd730c0dc9ddcf","Type":"ContainerStarted","Data":"a134e65260ac4d9787531b36788cd4bb39cd000005654ac56a919102c09b10eb"}
Jul 29 10:29:07.514235 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.514200    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k" event={"ID":"b579a599-b611-43a6-b50d-2e9b5a936c32","Type":"ContainerStarted","Data":"e887973e448926f1d5ec26de054e6ad9e8f3b40a8c89875d0109140e727362b3"}
Jul 29 10:29:07.515472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.515453    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gl58w" event={"ID":"2818e4ae-44ec-460b-a866-86b544e12a30","Type":"ContainerStarted","Data":"9eb40cc13d751ea1bf10b842dde4e11a1a266c3fb1472b829bb99eafa72702b3"}
Jul 29 10:29:07.517948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.517929    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"7a4e0527fefd1e08c60b037cb399e505ca32c05bf660bc8d1a41c2e35871b293"}
Jul 29 10:29:07.518057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.517999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"5fc3331c6d846a30b3c737127f0d465e1b756a6398211af9d64f220f46421458"}
Jul 29 10:29:07.518057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.518051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"a0f85f285fe3c3932bfc1e5f3e442ccc30fc01c8694043395495b53e5df2ce14"}
Jul 29 10:29:07.518168 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.518100    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"71187d198de007501333772eaf5d322acd31528fd3fe2f63059bdc3e26ef4864"}
Jul 29 10:29:07.518168 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.518151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"62c6ffa9ae6745213c20d966603c15317b8e94ef30d9a89dac47440ebdb31f0e"}
Jul 29 10:29:07.557445 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.557385    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9ncwb" podStartSLOduration=1.361125501 podStartE2EDuration="22.557366472s" totalImagesPullingTime="21.196240971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.541478395 +0000 UTC m=+23.721860275" watchObservedRunningTime="2026-07-29 10:29:07.557366472 +0000 UTC m=+23.737748352"
Jul 29 10:29:07.578547 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.578487    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-86.ec2.internal" podStartSLOduration=22.578470403 podStartE2EDuration="22.578470403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.577977098 +0000 UTC m=+23.758358978" watchObservedRunningTime="2026-07-29 10:29:07.578470403 +0000 UTC m=+23.758852282"
Jul 29 10:29:07.578701 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.578557    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xkgp2" podStartSLOduration=1.473641127 podStartE2EDuration="22.578551291s" totalImagesPullingTime="21.104910164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.563823909 +0000 UTC m=+23.744205779" watchObservedRunningTime="2026-07-29 10:29:07.578551291 +0000 UTC m=+23.758933167"
Jul 29 10:29:07.599074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.599024    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dsdrb" podStartSLOduration=2.404716199 podStartE2EDuration="23.599009071s" totalImagesPullingTime="21.194292872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.598566251 +0000 UTC m=+23.778948131" watchObservedRunningTime="2026-07-29 10:29:07.599009071 +0000 UTC m=+23.779390950"
Jul 29 10:29:07.623593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.623544    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6nvz2" podStartSLOduration=2.447924989 podStartE2EDuration="23.623526091s" totalImagesPullingTime="21.175601102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.623007387 +0000 UTC m=+23.803389268" watchObservedRunningTime="2026-07-29 10:29:07.623526091 +0000 UTC m=+23.803907972"
Jul 29 10:29:07.638740 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:07.638690    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gl58w" podStartSLOduration=1.499740883 podStartE2EDuration="22.63867021s" totalImagesPullingTime="21.138929327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:07.638510386 +0000 UTC m=+23.818892267" watchObservedRunningTime="2026-07-29 10:29:07.63867021 +0000 UTC m=+23.819052090"
Jul 29 10:29:08.469824 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:08.469340    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 10:29:08.523007 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:08.522964    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k" event={"ID":"b579a599-b611-43a6-b50d-2e9b5a936c32","Type":"ContainerStarted","Data":"deb437032fe0ce95c347851159478c571ea83b51a1a099d4335e29e153af9dad"}
Jul 29 10:29:08.526695 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:08.526659    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"81262db80e460a48a689aa5a8bf3672cf0241760a52c9902a370767521b6ef27"}
Jul 29 10:29:08.528446 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:08.528410    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqvvn" event={"ID":"e1c66da3-c1dc-4e85-9ae2-79ae5c1db565","Type":"ContainerStarted","Data":"49f88dc588901834a427f3213b811fbb7001207f3da6de5d7942d77df986da85"}
Jul 29 10:29:09.287908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.287802    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T10:29:08.469361836Z","UUID":"07016ff8-b258-47ed-9f8d-c626242b9af6","Handler":null,"Name":"","Endpoint":""}
Jul 29 10:29:09.289345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.289317    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 10:29:09.289505 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.289365    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 10:29:09.336056 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.336015    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:09.336257 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:09.336122    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:09.336582 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.336562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:09.336708 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:09.336655    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:09.336708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.336683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:09.336830 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:09.336811    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:09.532943 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.532900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal" event={"ID":"f6e2c422f24a43001d363c82a2d67531","Type":"ContainerStarted","Data":"29468aeed6cd74d8662a2309360e3a2f904d6428d3e18a1a3fadc135757e8fb1"}
Jul 29 10:29:09.546707 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.546615    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-86.ec2.internal" podStartSLOduration=24.546599311 podStartE2EDuration="24.546599311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:09.546250605 +0000 UTC m=+25.726632482" watchObservedRunningTime="2026-07-29 10:29:09.546599311 +0000 UTC m=+25.726981192"
Jul 29 10:29:09.546879 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.546832    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bqvvn" podStartSLOduration=4.382156835 podStartE2EDuration="25.54682162s" totalImagesPullingTime="21.164664785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:08.557932605 +0000 UTC m=+24.738314484" watchObservedRunningTime="2026-07-29 10:29:09.54682162 +0000 UTC m=+25.727203501"
Jul 29 10:29:09.625896 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.625874    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:29:09.626434 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:09.626389    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:29:10.539535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:10.539415    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k" event={"ID":"b579a599-b611-43a6-b50d-2e9b5a936c32","Type":"ContainerStarted","Data":"6a5980fd540186baa6d254e974b7d0db8797c3b18c24c3e833f3920aaf2ab405"}
Jul 29 10:29:10.544028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:10.543979    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"9945f77516a82ac3e70c84db63a5e5d244ab00f534644cd2822db37bfd23415d"}
Jul 29 10:29:10.544286 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:10.544083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:29:10.544899 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:10.544872    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dsdrb"
Jul 29 10:29:10.557795 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:10.557732    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hl6k" podStartSLOduration=2.268659081 podStartE2EDuration="25.557712209s" totalImagesPullingTime="23.289053128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:10.557351668 +0000 UTC m=+26.737733548" watchObservedRunningTime="2026-07-29 10:29:10.557712209 +0000 UTC m=+26.738094088"
Jul 29 10:29:11.336124 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:11.336084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:11.336124 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:11.336122    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:11.336413 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:11.336083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:11.336413 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:11.336261    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:11.336413 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:11.336327    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:11.336524 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:11.336426    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:12.552954 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.552707    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" event={"ID":"642a5976-8b42-47d0-a96b-7885f7385cf6","Type":"ContainerStarted","Data":"d15eb3deeb946ce3d1a9dd5b990535c1482ef3d089de0cff6aa3a9363f25f4ad"}
Jul 29 10:29:12.553840 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.552958    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:12.554638 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.554613    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="9f311eb1739711ad152d2567c3733fef11f66b55bc70411cd3eb2307e46073be" exitCode=0
Jul 29 10:29:12.554724 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.554689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"9f311eb1739711ad152d2567c3733fef11f66b55bc70411cd3eb2307e46073be"}
Jul 29 10:29:12.569936 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.569907    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:12.578244 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:12.578180    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" podStartSLOduration=7.332574092 podStartE2EDuration="28.578165475s" totalImagesPullingTime="21.245591383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:12.578059631 +0000 UTC m=+28.758441512" watchObservedRunningTime="2026-07-29 10:29:12.578165475 +0000 UTC m=+28.758547355"
Jul 29 10:29:13.336082 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.335898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:13.336351 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.335905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:13.336351 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.335945    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:13.336351 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.336202    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:13.336351 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.336301    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:13.336563 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.336401    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:13.556978 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.556956    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 10:29:13.557443 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.557423    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:13.575905 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.575869    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:13.914404 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.914330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dsndc"]
Jul 29 10:29:13.914618 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.914452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:13.914618 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.914555    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:13.916932 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.916904    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5fm96"]
Jul 29 10:29:13.917086 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.917027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:13.917150 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.917119    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:13.917561 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.917540    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrh9c"]
Jul 29 10:29:13.917664 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:13.917640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:13.917753 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:13.917733    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:14.561188 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:14.561100    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="969d4c3cfac9ec44002b14db8803ccb176a9100242dfcaec450a28e5721bcfac" exitCode=0
Jul 29 10:29:14.561617 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:14.561241    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"969d4c3cfac9ec44002b14db8803ccb176a9100242dfcaec450a28e5721bcfac"}
Jul 29 10:29:14.561617 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:14.561270    2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 29 10:29:14.715851 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:14.715819    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:15.335845 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:15.335802    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:15.336169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:15.335806    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:15.336169 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:15.335948    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:15.336169 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:15.335996    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:16.336847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:16.336771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:16.337255 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:16.336921    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:16.568489 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:16.568444    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="7ea0296dd9cbb8e1ba78880271206ef15414a80eff720faa804ad0e2fd21a676" exitCode=0
Jul 29 10:29:16.568665 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:16.568515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"7ea0296dd9cbb8e1ba78880271206ef15414a80eff720faa804ad0e2fd21a676"}
Jul 29 10:29:16.584080 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:16.584032    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m" podUID="642a5976-8b42-47d0-a96b-7885f7385cf6" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 29 10:29:17.069344 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:17.069244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:17.069344 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:17.069328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:17.069576 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.069393    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.069576 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.069449    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.069576 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.069467    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret podName:78f23bd5-32c5-4c6e-a8b1-a72745649bcd nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.069448178 +0000 UTC m=+65.249830036 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret") pod "global-pull-secret-syncer-dsndc" (UID: "78f23bd5-32c5-4c6e-a8b1-a72745649bcd") : object "kube-system"/"original-pull-secret" not registered
Jul 29 10:29:17.069576 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.069503    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.06948663 +0000 UTC m=+65.249868490 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 10:29:17.169791 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:17.169756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:17.169967 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.169926    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 10:29:17.169967 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.169951    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 10:29:17.169967 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.169961    2574 projected.go:196] Error preparing data for projected volume kube-api-access-pg88d for pod openshift-network-diagnostics/network-check-target-mrh9c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:17.170068 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.170014    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d podName:5b918d79-6db9-4714-9e61-49f1e2460478 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:49.16999966 +0000 UTC m=+65.350381521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg88d" (UniqueName: "kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d") pod "network-check-target-mrh9c" (UID: "5b918d79-6db9-4714-9e61-49f1e2460478") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 10:29:17.336640 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:17.336538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:17.336640 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:17.336549    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:17.336872 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.336668    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:17.336872 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:17.336725    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:18.336809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:18.336526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:18.338089 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:18.338058    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dsndc" podUID="78f23bd5-32c5-4c6e-a8b1-a72745649bcd"
Jul 29 10:29:19.335962 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.335930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:19.336130 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.335976    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:19.336130 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.336080    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5fm96" podUID="2df1e671-4dab-4eed-b983-326d3fc53b9c"
Jul 29 10:29:19.336285 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.336245    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrh9c" podUID="5b918d79-6db9-4714-9e61-49f1e2460478"
Jul 29 10:29:19.644464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.644378    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-86.ec2.internal" event="NodeReady"
Jul 29 10:29:19.644916 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.644537    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 10:29:19.679267 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.678967    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:29:19.681739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.681708    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.683854 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.683830    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 10:29:19.689750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.689699    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g6k9\""
Jul 29 10:29:19.690908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.690889    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 10:29:19.694611 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.694591    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 10:29:19.696882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.696848    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z2cth"]
Jul 29 10:29:19.698899 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.698879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.700975 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.700956    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 10:29:19.701663 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.701642    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 10:29:19.701955 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.701941    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:29:19.714321 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.714262    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8tqhw"]
Jul 29 10:29:19.717888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.717863    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 10:29:19.718258 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.718209    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:29:19.718493 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.718439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.719112 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.719092    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2cth"]
Jul 29 10:29:19.723019 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.723000    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 10:29:19.724184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.724004    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 10:29:19.724184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.724051    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 10:29:19.724356 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.724339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nxd67\""
Jul 29 10:29:19.738310 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.738284    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8tqhw"]
Jul 29 10:29:19.790010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.789981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cd63fd-0295-493e-a581-d3af51c97345-cert\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.790010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh6ch\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790312    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df56f12f-8167-4d7b-8863-7a21eeba4d97-config-volume\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790338    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvsg\" (UniqueName: \"kubernetes.io/projected/e0cd63fd-0295-493e-a581-d3af51c97345-kube-api-access-rtvsg\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df56f12f-8167-4d7b-8863-7a21eeba4d97-tmp-dir\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sks8\" (UniqueName: \"kubernetes.io/projected/df56f12f-8167-4d7b-8863-7a21eeba4d97-kube-api-access-6sks8\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.790497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.790478    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.891434 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891396    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.891434 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cd63fd-0295-493e-a581-d3af51c97345-cert\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891528    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh6ch\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.891602    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:19.891682 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.891679    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:20.391658795 +0000 UTC m=+36.572040667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:19.891954 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892016 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.891989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892079 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892079 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df56f12f-8167-4d7b-8863-7a21eeba4d97-config-volume\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.892184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvsg\" (UniqueName: \"kubernetes.io/projected/e0cd63fd-0295-493e-a581-d3af51c97345-kube-api-access-rtvsg\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.892184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df56f12f-8167-4d7b-8863-7a21eeba4d97-tmp-dir\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.892364 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892199    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sks8\" (UniqueName: \"kubernetes.io/projected/df56f12f-8167-4d7b-8863-7a21eeba4d97-kube-api-access-6sks8\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.892364 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892782 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.892935 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.893029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892914    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df56f12f-8167-4d7b-8863-7a21eeba4d97-config-volume\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.893029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.892978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df56f12f-8167-4d7b-8863-7a21eeba4d97-tmp-dir\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.893168 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.893030    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:19.893168 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.893045    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:19.893168 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:19.893099    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:20.393083738 +0000 UTC m=+36.573465608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:19.897122 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.897056    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.897122 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.897063    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0cd63fd-0295-493e-a581-d3af51c97345-cert\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.897122 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.897062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.900868 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.900841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvsg\" (UniqueName: \"kubernetes.io/projected/e0cd63fd-0295-493e-a581-d3af51c97345-kube-api-access-rtvsg\") pod \"ingress-canary-8tqhw\" (UID: \"e0cd63fd-0295-493e-a581-d3af51c97345\") " pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:19.901049 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.901028    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:19.901168 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.901142    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sks8\" (UniqueName: \"kubernetes.io/projected/df56f12f-8167-4d7b-8863-7a21eeba4d97-kube-api-access-6sks8\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:19.901633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:19.901612    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh6ch\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:20.028535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:20.028489    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8tqhw"
Jul 29 10:29:20.336415 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:20.336339    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:20.338914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:20.338894    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:20.397100 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:20.397056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:20.397296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:20.397125    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:20.397296 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:20.397238    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:20.397296 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:20.397257    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:20.397296 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:20.397269    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:20.397468 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:20.397320    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:21.397302102 +0000 UTC m=+37.577683960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:20.397468 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:20.397335    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:21.397327801 +0000 UTC m=+37.577709661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.051425 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.051269    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8tqhw"]
Jul 29 10:29:21.055190 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:21.055159    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cd63fd_0295_493e_a581_d3af51c97345.slice/crio-b0ab11be7c5c7b6de0c399d25828965760bb9a962a1e16fb7f95bd568fbfb335 WatchSource:0}: Error finding container b0ab11be7c5c7b6de0c399d25828965760bb9a962a1e16fb7f95bd568fbfb335: Status 404 returned error can't find the container with id b0ab11be7c5c7b6de0c399d25828965760bb9a962a1e16fb7f95bd568fbfb335
Jul 29 10:29:21.336234 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.336184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:21.336430 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.336409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:21.338602 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.338580    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 10:29:21.339090 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.339073    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:21.339183 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.339102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:29:21.339183 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.339099    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 10:29:21.339365 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.339102    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:21.404750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.404712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:21.404928 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.404774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:21.404928 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:21.404871    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:21.404928 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:21.404881    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:21.404928 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:21.404873    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.405079 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:21.404925    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:23.404911388 +0000 UTC m=+39.585293245 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:21.405079 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:21.404975    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:23.404963592 +0000 UTC m=+39.585345451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:21.584969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.584875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8tqhw" event={"ID":"e0cd63fd-0295-493e-a581-d3af51c97345","Type":"ContainerStarted","Data":"b0ab11be7c5c7b6de0c399d25828965760bb9a962a1e16fb7f95bd568fbfb335"}
Jul 29 10:29:21.588268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.588233    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="436600b501e55ee20b02f41f7bd5ccd895288a5dab48b1bbe9c07c301f194e12" exitCode=0
Jul 29 10:29:21.588407 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:21.588331    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"436600b501e55ee20b02f41f7bd5ccd895288a5dab48b1bbe9c07c301f194e12"}
Jul 29 10:29:22.595596 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:22.595556    2574 generic.go:356] "Generic (PLEG): container finished" podID="5442ddcf-c235-46d4-a3bb-faace9f9f281" containerID="385c706e3bd40ded15588a798506fd2d62c450664a2a721381cdf8bb54f5d95f" exitCode=0
Jul 29 10:29:22.596062 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:22.595624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerDied","Data":"385c706e3bd40ded15588a798506fd2d62c450664a2a721381cdf8bb54f5d95f"}
Jul 29 10:29:23.423709 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:23.423602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:23.423709 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:23.423674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:23.423890 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:23.423787    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:23.423890 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:23.423817    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:23.423890 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:23.423831    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:23.423890 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:23.423873    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:27.42385243 +0000 UTC m=+43.604234306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:23.423890 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:23.423887    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:27.423881577 +0000 UTC m=+43.604263435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:23.602071 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:23.602038    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lk64z" event={"ID":"5442ddcf-c235-46d4-a3bb-faace9f9f281","Type":"ContainerStarted","Data":"8972682a613041e304a2c70de43434c43497bccb7a8982ca2417d3303d284835"}
Jul 29 10:29:23.623618 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:23.623564    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lk64z" podStartSLOduration=6.638629347 podStartE2EDuration="38.623544227s" totalImagesPullingTime="31.98491488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:23.622279193 +0000 UTC m=+39.802661074" watchObservedRunningTime="2026-07-29 10:29:23.623544227 +0000 UTC m=+39.803926107"
Jul 29 10:29:24.605652 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:24.605406    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8tqhw" event={"ID":"e0cd63fd-0295-493e-a581-d3af51c97345","Type":"ContainerStarted","Data":"d6c24e5857afdccd8b82ddeadffb1f2eba045b36c2f42fc0581cfa9d7d3fd7b8"}
Jul 29 10:29:24.621788 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:24.621739    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8tqhw" podStartSLOduration=3.148918965 podStartE2EDuration="5.62172307s" totalImagesPullingTime="2.472804105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:24.621350003 +0000 UTC m=+40.801731883" watchObservedRunningTime="2026-07-29 10:29:24.62172307 +0000 UTC m=+40.802104980"
Jul 29 10:29:27.454118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:27.454072    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:27.454142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:27.454253    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:27.454286    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:27.454297    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:27.454323    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:35.45430674 +0000 UTC m=+51.634688602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:27.454495 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:27.454340    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:35.454329243 +0000 UTC m=+51.634711133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:32.554642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.554612    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bmxdx"]
Jul 29 10:29:32.565154 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.565126    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.567015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.566990    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bmxdx"]
Jul 29 10:29:32.568268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.568244    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.569388 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.569364    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rcqmd\""
Jul 29 10:29:32.569490 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.569417    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.570337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.570024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 29 10:29:32.570337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.570163    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 29 10:29:32.573844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.573768    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 29 10:29:32.642344 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.642303    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"]
Jul 29 10:29:32.660327 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.660298    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.662107 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.662083    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"]
Jul 29 10:29:32.662473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.662455    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:29:32.662732 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.662703    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d6qsl\""
Jul 29 10:29:32.662859 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.662774    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 10:29:32.662859 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.662778    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 10:29:32.663301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.663278    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 10:29:32.692778 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.692743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20188e15-550f-46cd-b99b-a5bf59756d5b-serving-cert\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.692946 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.692875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-trusted-ca\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.692946 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.692916    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-config\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.692946 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.692941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfmq\" (UniqueName: \"kubernetes.io/projected/20188e15-550f-46cd-b99b-a5bf59756d5b-kube-api-access-wjfmq\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.745703 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.745669    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"]
Jul 29 10:29:32.759184 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.759150    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"]
Jul 29 10:29:32.759339 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.759304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"
Jul 29 10:29:32.761790 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.761763    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wj2cc\""
Jul 29 10:29:32.775903 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.775876    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"]
Jul 29 10:29:32.775903 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.775910    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"]
Jul 29 10:29:32.776066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.775993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:32.780125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.780097    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 10:29:32.780286 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.780156    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wm6pt\""
Jul 29 10:29:32.780478 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.780463    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 10:29:32.794169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-trusted-ca\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.794169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-config\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.794436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95ec880c-ff89-4c96-91d7-a390b557c278-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.794436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfmq\" (UniqueName: \"kubernetes.io/projected/20188e15-550f-46cd-b99b-a5bf59756d5b-kube-api-access-wjfmq\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.794436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20188e15-550f-46cd-b99b-a5bf59756d5b-serving-cert\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.794436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95ec880c-ff89-4c96-91d7-a390b557c278-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.794436 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjm6\" (UniqueName: \"kubernetes.io/projected/95ec880c-ff89-4c96-91d7-a390b557c278-kube-api-access-4qjm6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.795017 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.794989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-config\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.795127 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.795085    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/20188e15-550f-46cd-b99b-a5bf59756d5b-trusted-ca\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.798559 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.798540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20188e15-550f-46cd-b99b-a5bf59756d5b-serving-cert\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.814293 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.814210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfmq\" (UniqueName: \"kubernetes.io/projected/20188e15-550f-46cd-b99b-a5bf59756d5b-kube-api-access-wjfmq\") pod \"console-operator-584dcb6b54-bmxdx\" (UID: \"20188e15-550f-46cd-b99b-a5bf59756d5b\") " pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.878944 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.878908    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:32.895340 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrczt\" (UniqueName: \"kubernetes.io/projected/91d0a9bf-1cad-486e-a4d1-0fe99a060b10-kube-api-access-jrczt\") pod \"network-check-source-7d88cb55d8-kz24q\" (UID: \"91d0a9bf-1cad-486e-a4d1-0fe99a060b10\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"
Jul 29 10:29:32.895501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/825761a1-4047-4058-b59b-24871c8e45e5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:32.895501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:32.895626 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95ec880c-ff89-4c96-91d7-a390b557c278-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.895626 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895550    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95ec880c-ff89-4c96-91d7-a390b557c278-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.895626 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.895600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjm6\" (UniqueName: \"kubernetes.io/projected/95ec880c-ff89-4c96-91d7-a390b557c278-kube-api-access-4qjm6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.896114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.896095    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/95ec880c-ff89-4c96-91d7-a390b557c278-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.898510 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.898482    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/95ec880c-ff89-4c96-91d7-a390b557c278-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.903414 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.903386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjm6\" (UniqueName: \"kubernetes.io/projected/95ec880c-ff89-4c96-91d7-a390b557c278-kube-api-access-4qjm6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb\" (UID: \"95ec880c-ff89-4c96-91d7-a390b557c278\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.970245 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.970194    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"
Jul 29 10:29:32.996706 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.996672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrczt\" (UniqueName: \"kubernetes.io/projected/91d0a9bf-1cad-486e-a4d1-0fe99a060b10-kube-api-access-jrczt\") pod \"network-check-source-7d88cb55d8-kz24q\" (UID: \"91d0a9bf-1cad-486e-a4d1-0fe99a060b10\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"
Jul 29 10:29:32.996884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.996723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/825761a1-4047-4058-b59b-24871c8e45e5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:32.996954 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.996868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:32.997013 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:32.996974    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:32.997535 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:32.997030    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:33.497012967 +0000 UTC m=+49.677394825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:32.997535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:32.997427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/825761a1-4047-4058-b59b-24871c8e45e5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:33.007005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.006975    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrczt\" (UniqueName: \"kubernetes.io/projected/91d0a9bf-1cad-486e-a4d1-0fe99a060b10-kube-api-access-jrczt\") pod \"network-check-source-7d88cb55d8-kz24q\" (UID: \"91d0a9bf-1cad-486e-a4d1-0fe99a060b10\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"
Jul 29 10:29:33.026508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.026477    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bmxdx"]
Jul 29 10:29:33.030023 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:33.029996    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20188e15_550f_46cd_b99b_a5bf59756d5b.slice/crio-dfe5c875723c03327ab6f4ee9e0e9db388c1afc3c38d000fe557fff521977493 WatchSource:0}: Error finding container dfe5c875723c03327ab6f4ee9e0e9db388c1afc3c38d000fe557fff521977493: Status 404 returned error can't find the container with id dfe5c875723c03327ab6f4ee9e0e9db388c1afc3c38d000fe557fff521977493
Jul 29 10:29:33.068561 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.068473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"
Jul 29 10:29:33.140667 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.140622    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb"]
Jul 29 10:29:33.224501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.224472    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q"]
Jul 29 10:29:33.228155 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:33.228126    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d0a9bf_1cad_486e_a4d1_0fe99a060b10.slice/crio-e8a2ef1ff0efcea24ccaef6cdc7cd45d2f53cbe62c30d25a867e95acde6c9dad WatchSource:0}: Error finding container e8a2ef1ff0efcea24ccaef6cdc7cd45d2f53cbe62c30d25a867e95acde6c9dad: Status 404 returned error can't find the container with id e8a2ef1ff0efcea24ccaef6cdc7cd45d2f53cbe62c30d25a867e95acde6c9dad
Jul 29 10:29:33.500285 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.500176    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:33.500439 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:33.500339    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:33.500439 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:33.500416    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:34.500398895 +0000 UTC m=+50.680780760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:33.628088 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.628046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q" event={"ID":"91d0a9bf-1cad-486e-a4d1-0fe99a060b10","Type":"ContainerStarted","Data":"e8a2ef1ff0efcea24ccaef6cdc7cd45d2f53cbe62c30d25a867e95acde6c9dad"}
Jul 29 10:29:33.629469 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.629440    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" event={"ID":"20188e15-550f-46cd-b99b-a5bf59756d5b","Type":"ContainerStarted","Data":"dfe5c875723c03327ab6f4ee9e0e9db388c1afc3c38d000fe557fff521977493"}
Jul 29 10:29:33.630588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:33.630561    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb" event={"ID":"95ec880c-ff89-4c96-91d7-a390b557c278","Type":"ContainerStarted","Data":"2782cf77dbe4f0e32e283030435e580a6da1cac5006fb17edf25162e8257d371"}
Jul 29 10:29:34.508272 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:34.508205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:34.508474 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:34.508420    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:34.508540 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:34.508505    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:36.508481804 +0000 UTC m=+52.688863679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:35.518150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:35.518108    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:35.518186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:35.518266    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:35.518329    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:35.518345    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65bb9b8556-g49sd: secret "image-registry-tls" not found
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:35.518345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls podName:df56f12f-8167-4d7b-8863-7a21eeba4d97 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:51.518325278 +0000 UTC m=+67.698707152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls") pod "dns-default-z2cth" (UID: "df56f12f-8167-4d7b-8863-7a21eeba4d97") : secret "dns-default-metrics-tls" not found
Jul 29 10:29:35.518594 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:35.518411    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls podName:453a394d-7d32-490a-9ac2-3be63bb6d98d nodeName:}" failed. No retries permitted until 2026-07-29 10:29:51.518398371 +0000 UTC m=+67.698780232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls") pod "image-registry-65bb9b8556-g49sd" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d") : secret "image-registry-tls" not found
Jul 29 10:29:36.528204 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.528161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:36.528676 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:36.528323    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:36.528676 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:36.528381    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.528366504 +0000 UTC m=+56.708748362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:36.892973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.892889    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q6pkk"]
Jul 29 10:29:36.926051 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.926017    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:36.928610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.928575    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 10:29:36.928832 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.928811    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 10:29:36.929118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.929092    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 10:29:36.929291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.929264    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 10:29:36.929496 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.929477    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:29:36.933620 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:36.933596    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6pkk"]
Jul 29 10:29:37.033715 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.033677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e233220-65a3-4899-b378-8fbccc025d1a-crio-socket\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.033715 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.033711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e233220-65a3-4899-b378-8fbccc025d1a-data-volume\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.033914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.033742    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.033914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.033844    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.033914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.033880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlzt\" (UniqueName: \"kubernetes.io/projected/5e233220-65a3-4899-b378-8fbccc025d1a-kube-api-access-grlzt\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e233220-65a3-4899-b378-8fbccc025d1a-crio-socket\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e233220-65a3-4899-b378-8fbccc025d1a-data-volume\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134579    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlzt\" (UniqueName: \"kubernetes.io/projected/5e233220-65a3-4899-b378-8fbccc025d1a-kube-api-access-grlzt\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5e233220-65a3-4899-b378-8fbccc025d1a-crio-socket\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.134809 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.134745    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.134809 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.134767    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.134953 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.134818    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.634799601 +0000 UTC m=+53.815181475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.134953 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.134834    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:37.634826499 +0000 UTC m=+53.815208357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.134953 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.134847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e233220-65a3-4899-b378-8fbccc025d1a-data-volume\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.142754 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.142734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlzt\" (UniqueName: \"kubernetes.io/projected/5e233220-65a3-4899-b378-8fbccc025d1a-kube-api-access-grlzt\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.640122 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.640090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.640605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.640167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:37.640605 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.640246    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.640605 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.640264    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.640605 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.640318    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.640295395 +0000 UTC m=+54.820677268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:37.640605 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:37.640336    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:38.640327528 +0000 UTC m=+54.820709385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:37.643736 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.643324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb" event={"ID":"95ec880c-ff89-4c96-91d7-a390b557c278","Type":"ContainerStarted","Data":"710ae889a7e6813fc69cf72ac98988f555100caf32f3a84461797d588107f3bf"}
Jul 29 10:29:37.645925 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.645293    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q" event={"ID":"91d0a9bf-1cad-486e-a4d1-0fe99a060b10","Type":"ContainerStarted","Data":"67bdfd4c321b24cd29ff6fd93330e28becfe3c819ad702177545f3afa7adb837"}
Jul 29 10:29:37.647109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.647089    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/0.log"
Jul 29 10:29:37.647233 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.647129    2574 generic.go:356] "Generic (PLEG): container finished" podID="20188e15-550f-46cd-b99b-a5bf59756d5b" containerID="4614ca328d14ff13f240ef4cb79fda7a59babdadbb08679891594a00a5c773a1" exitCode=255
Jul 29 10:29:37.647233 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.647164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" event={"ID":"20188e15-550f-46cd-b99b-a5bf59756d5b","Type":"ContainerDied","Data":"4614ca328d14ff13f240ef4cb79fda7a59babdadbb08679891594a00a5c773a1"}
Jul 29 10:29:37.647488 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.647472    2574 scope.go:117] "RemoveContainer" containerID="4614ca328d14ff13f240ef4cb79fda7a59babdadbb08679891594a00a5c773a1"
Jul 29 10:29:37.656992 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.656946    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb" podStartSLOduration=1.4182465610000001 podStartE2EDuration="5.656929821s" totalImagesPullingTime="4.23868326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:37.656348219 +0000 UTC m=+53.836730101" watchObservedRunningTime="2026-07-29 10:29:37.656929821 +0000 UTC m=+53.837311755"
Jul 29 10:29:37.689324 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:37.688091    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-kz24q" podStartSLOduration=1.521588449 podStartE2EDuration="5.688067861s" totalImagesPullingTime="4.166479412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:37.686186366 +0000 UTC m=+53.866568247" watchObservedRunningTime="2026-07-29 10:29:37.688067861 +0000 UTC m=+53.868449742"
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.649957    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.650023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:38.650099    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:38.650104    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:38.650162    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.650143721 +0000 UTC m=+56.830525580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:38.650259 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:38.650179    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:40.650170142 +0000 UTC m=+56.830552006 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:38.669914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.669886    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:29:38.670365 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.670346    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/0.log"
Jul 29 10:29:38.670479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.670382    2574 generic.go:356] "Generic (PLEG): container finished" podID="20188e15-550f-46cd-b99b-a5bf59756d5b" containerID="c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4" exitCode=255
Jul 29 10:29:38.670517 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.670473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" event={"ID":"20188e15-550f-46cd-b99b-a5bf59756d5b","Type":"ContainerDied","Data":"c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4"}
Jul 29 10:29:38.670551 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.670534    2574 scope.go:117] "RemoveContainer" containerID="4614ca328d14ff13f240ef4cb79fda7a59babdadbb08679891594a00a5c773a1"
Jul 29 10:29:38.670824 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:38.670802    2574 scope.go:117] "RemoveContainer" containerID="c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4"
Jul 29 10:29:38.670989 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:38.670966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bmxdx_openshift-console-operator(20188e15-550f-46cd-b99b-a5bf59756d5b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" podUID="20188e15-550f-46cd-b99b-a5bf59756d5b"
Jul 29 10:29:39.040821 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:39.040792    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gl58w_2818e4ae-44ec-460b-a866-86b544e12a30/dns-node-resolver/0.log"
Jul 29 10:29:39.640182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:39.640155    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6nvz2_a6c58d40-a2da-4b45-aa50-091fe6d34cca/node-ca/0.log"
Jul 29 10:29:39.674628 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:39.674605    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:29:39.675010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:39.674958    2574 scope.go:117] "RemoveContainer" containerID="c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4"
Jul 29 10:29:39.675167 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:39.675147    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bmxdx_openshift-console-operator(20188e15-550f-46cd-b99b-a5bf59756d5b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" podUID="20188e15-550f-46cd-b99b-a5bf59756d5b"
Jul 29 10:29:40.566083 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:40.566038    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:40.566268 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.566191    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:40.566309 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.566288    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:29:48.566270796 +0000 UTC m=+64.746652658 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:40.642728 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:40.642683    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8tqhw_e0cd63fd-0295-493e-a581-d3af51c97345/serve-healthcheck-canary/0.log"
Jul 29 10:29:40.667131 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:40.667086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:40.667269 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:40.667152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:40.667269 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.667256    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:40.667399 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.667314    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:40.667399 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.667320    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.66730572 +0000 UTC m=+60.847687577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:40.667399 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:40.667391    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:44.667372561 +0000 UTC m=+60.847754437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:42.879798 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:42.879762    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:42.880291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:42.879820    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:42.880353 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:42.880310    2574 scope.go:117] "RemoveContainer" containerID="c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4"
Jul 29 10:29:42.880526 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:42.880503    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bmxdx_openshift-console-operator(20188e15-550f-46cd-b99b-a5bf59756d5b)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" podUID="20188e15-550f-46cd-b99b-a5bf59756d5b"
Jul 29 10:29:44.700488 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:44.700450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:44.700930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:44.700566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:44.700930 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:44.700575    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 10:29:44.700930 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:44.700638    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.700621674 +0000 UTC m=+68.881003532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : configmap "kube-rbac-proxy" not found
Jul 29 10:29:44.700930 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:44.700665    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:44.700930 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:44.700711    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls podName:5e233220-65a3-4899-b378-8fbccc025d1a nodeName:}" failed. No retries permitted until 2026-07-29 10:29:52.70069393 +0000 UTC m=+68.881075806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q6pkk" (UID: "5e233220-65a3-4899-b378-8fbccc025d1a") : secret "insights-runtime-extractor-tls" not found
Jul 29 10:29:46.579861 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:46.579821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jkg9m"
Jul 29 10:29:48.633330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:48.633282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:29:48.633820 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:48.633464    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 10:29:48.633820 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:48.633557    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert podName:825761a1-4047-4058-b59b-24871c8e45e5 nodeName:}" failed. No retries permitted until 2026-07-29 10:30:04.633535807 +0000 UTC m=+80.813917665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-cdrjg" (UID: "825761a1-4047-4058-b59b-24871c8e45e5") : secret "networking-console-plugin-cert" not found
Jul 29 10:29:49.137488 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.137432    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:29:49.137706 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.137503    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:49.139639 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.139601    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 10:29:49.139639 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.139604    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 10:29:49.148294 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:49.148262    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 10:29:49.148382 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:29:49.148346    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs podName:2df1e671-4dab-4eed-b983-326d3fc53b9c nodeName:}" failed. No retries permitted until 2026-07-29 10:30:53.148323756 +0000 UTC m=+129.328705627 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs") pod "network-metrics-daemon-5fm96" (UID: "2df1e671-4dab-4eed-b983-326d3fc53b9c") : secret "metrics-daemon-secret" not found
Jul 29 10:29:49.150660 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.150636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78f23bd5-32c5-4c6e-a8b1-a72745649bcd-original-pull-secret\") pod \"global-pull-secret-syncer-dsndc\" (UID: \"78f23bd5-32c5-4c6e-a8b1-a72745649bcd\") " pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:49.237850 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.237813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:49.240725 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.240700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg88d\" (UniqueName: \"kubernetes.io/projected/5b918d79-6db9-4714-9e61-49f1e2460478-kube-api-access-pg88d\") pod \"network-check-target-mrh9c\" (UID: \"5b918d79-6db9-4714-9e61-49f1e2460478\") " pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:49.253067 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.253045    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h2t8s\""
Jul 29 10:29:49.261584 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.261569    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:49.392210 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.392133    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrh9c"]
Jul 29 10:29:49.395543 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:49.395515    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b918d79_6db9_4714_9e61_49f1e2460478.slice/crio-76bc7d6ef1e0f9637deb2cca05d210af7c47581e1b713087222a926f3ed87db1 WatchSource:0}: Error finding container 76bc7d6ef1e0f9637deb2cca05d210af7c47581e1b713087222a926f3ed87db1: Status 404 returned error can't find the container with id 76bc7d6ef1e0f9637deb2cca05d210af7c47581e1b713087222a926f3ed87db1
Jul 29 10:29:49.447671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.447646    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dsndc"
Jul 29 10:29:49.573238 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.573191    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dsndc"]
Jul 29 10:29:49.576753 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:49.576724    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f23bd5_32c5_4c6e_a8b1_a72745649bcd.slice/crio-929e935d7b41a0d3ba3c6221b16bd0ae7523d8ec34983832c032c5bcae302231 WatchSource:0}: Error finding container 929e935d7b41a0d3ba3c6221b16bd0ae7523d8ec34983832c032c5bcae302231: Status 404 returned error can't find the container with id 929e935d7b41a0d3ba3c6221b16bd0ae7523d8ec34983832c032c5bcae302231
Jul 29 10:29:49.699562 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.699471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dsndc" event={"ID":"78f23bd5-32c5-4c6e-a8b1-a72745649bcd","Type":"ContainerStarted","Data":"929e935d7b41a0d3ba3c6221b16bd0ae7523d8ec34983832c032c5bcae302231"}
Jul 29 10:29:49.700946 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.700923    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrh9c" event={"ID":"5b918d79-6db9-4714-9e61-49f1e2460478","Type":"ContainerStarted","Data":"480e56795b0ab3e5a63c780a2d216f8a7e031c190cb4c9b4462ff3b1cd9d18d1"}
Jul 29 10:29:49.701061 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.700961    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrh9c" event={"ID":"5b918d79-6db9-4714-9e61-49f1e2460478","Type":"ContainerStarted","Data":"76bc7d6ef1e0f9637deb2cca05d210af7c47581e1b713087222a926f3ed87db1"}
Jul 29 10:29:49.701099 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.701060    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:29:49.716038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:49.715987    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mrh9c" podStartSLOduration=64.715972011 podStartE2EDuration="1m4.715972011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:49.715479658 +0000 UTC m=+65.895861537" watchObservedRunningTime="2026-07-29 10:29:49.715972011 +0000 UTC m=+65.896353891"
Jul 29 10:29:51.558555 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.558512    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:51.559005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.558595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:51.561185 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.561157    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df56f12f-8167-4d7b-8863-7a21eeba4d97-metrics-tls\") pod \"dns-default-z2cth\" (UID: \"df56f12f-8167-4d7b-8863-7a21eeba4d97\") " pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:51.561296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.561255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"image-registry-65bb9b8556-g49sd\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") " pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:51.796779 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.796747    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8g6k9\""
Jul 29 10:29:51.805361 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.805312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:51.810580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.810512    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rdn4\""
Jul 29 10:29:51.819266 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:51.819230    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:52.771387 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.771337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:52.771920 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.771470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:52.772038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.772011    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5e233220-65a3-4899-b378-8fbccc025d1a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:52.774465 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.774436    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5e233220-65a3-4899-b378-8fbccc025d1a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q6pkk\" (UID: \"5e233220-65a3-4899-b378-8fbccc025d1a\") " pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:52.840454 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.840410    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l6k5g\""
Jul 29 10:29:52.849171 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:52.849130    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q6pkk"
Jul 29 10:29:53.254350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.254314    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2cth"]
Jul 29 10:29:53.257745 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:53.257716    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf56f12f_8167_4d7b_8863_7a21eeba4d97.slice/crio-7488ed798d10cac641e02274cb918ba5bcaa7127836c38e290ca5e85c85c2775 WatchSource:0}: Error finding container 7488ed798d10cac641e02274cb918ba5bcaa7127836c38e290ca5e85c85c2775: Status 404 returned error can't find the container with id 7488ed798d10cac641e02274cb918ba5bcaa7127836c38e290ca5e85c85c2775
Jul 29 10:29:53.480508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.480482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q6pkk"]
Jul 29 10:29:53.483889 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.483862    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:29:53.490063 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:53.490035    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453a394d_7d32_490a_9ac2_3be63bb6d98d.slice/crio-76cd37fe9ee3f8c517fd262af62b00e4786a63bcbc82ae1a2cbb7efc84405dd0 WatchSource:0}: Error finding container 76cd37fe9ee3f8c517fd262af62b00e4786a63bcbc82ae1a2cbb7efc84405dd0: Status 404 returned error can't find the container with id 76cd37fe9ee3f8c517fd262af62b00e4786a63bcbc82ae1a2cbb7efc84405dd0
Jul 29 10:29:53.717155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.717057    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6pkk" event={"ID":"5e233220-65a3-4899-b378-8fbccc025d1a","Type":"ContainerStarted","Data":"2601606d5b5eedba661ab3a95f265f04ab6641041bc0714d7258f89bc33a2832"}
Jul 29 10:29:53.719684 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.719647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" event={"ID":"453a394d-7d32-490a-9ac2-3be63bb6d98d","Type":"ContainerStarted","Data":"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"}
Jul 29 10:29:53.719865 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.719708    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" event={"ID":"453a394d-7d32-490a-9ac2-3be63bb6d98d","Type":"ContainerStarted","Data":"76cd37fe9ee3f8c517fd262af62b00e4786a63bcbc82ae1a2cbb7efc84405dd0"}
Jul 29 10:29:53.719981 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.719952    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:29:53.721572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.721538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2cth" event={"ID":"df56f12f-8167-4d7b-8863-7a21eeba4d97","Type":"ContainerStarted","Data":"7488ed798d10cac641e02274cb918ba5bcaa7127836c38e290ca5e85c85c2775"}
Jul 29 10:29:53.724826 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.723869    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dsndc" event={"ID":"78f23bd5-32c5-4c6e-a8b1-a72745649bcd","Type":"ContainerStarted","Data":"87474b4487ff02f0468f53ce7cfc69ec894d97be83bf8f4f461176b4e6251322"}
Jul 29 10:29:53.741852 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.741774    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" podStartSLOduration=68.741746641 podStartE2EDuration="1m8.741746641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:53.739817926 +0000 UTC m=+69.920199806" watchObservedRunningTime="2026-07-29 10:29:53.741746641 +0000 UTC m=+69.922128521"
Jul 29 10:29:53.754764 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:53.754686    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dsndc" podStartSLOduration=64.871382541 podStartE2EDuration="1m8.754663151s" totalImagesPullingTime="3.88328061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:53.754367227 +0000 UTC m=+69.934749108" watchObservedRunningTime="2026-07-29 10:29:53.754663151 +0000 UTC m=+69.935045031"
Jul 29 10:29:55.734932 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:55.734897    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2cth" event={"ID":"df56f12f-8167-4d7b-8863-7a21eeba4d97","Type":"ContainerStarted","Data":"568870f0caf73c7af24a6aae3f870153c09d345114a6eaf110520d7860bf60b4"}
Jul 29 10:29:55.735430 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:55.734954    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2cth" event={"ID":"df56f12f-8167-4d7b-8863-7a21eeba4d97","Type":"ContainerStarted","Data":"ca56bce38d2218df5014c59393e575d499a81a72a72ac750fd057636bfb5694f"}
Jul 29 10:29:55.735430 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:55.735044    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z2cth"
Jul 29 10:29:56.337163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:56.337119    2574 scope.go:117] "RemoveContainer" containerID="c8e0a0a2e1d630daf19c050e7efa82ebeda12807ddb7350d129f34cf5684c7e4"
Jul 29 10:29:59.119323 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.119200    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z2cth" podStartSLOduration=38.235793676 podStartE2EDuration="40.119180276s" totalImagesPullingTime="1.8833866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:55.752166926 +0000 UTC m=+71.932548807" watchObservedRunningTime="2026-07-29 10:29:59.119180276 +0000 UTC m=+75.299562155"
Jul 29 10:29:59.119683 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.119610    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"]
Jul 29 10:29:59.123565 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.123540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.125691 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.125663    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 10:29:59.126176 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.126159    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 10:29:59.126268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.126166    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w8sg9\""
Jul 29 10:29:59.127006 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.126988    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 10:29:59.127374 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.127357    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 10:29:59.130461 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.130408    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"]
Jul 29 10:29:59.233434 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.233390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n4j\" (UniqueName: \"kubernetes.io/projected/0cf8ad64-6646-4d3b-af99-53d142ed7a55-kube-api-access-k2n4j\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.233630 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.233528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf8ad64-6646-4d3b-af99-53d142ed7a55-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.335178 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.334691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf8ad64-6646-4d3b-af99-53d142ed7a55-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.335178 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.334814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n4j\" (UniqueName: \"kubernetes.io/projected/0cf8ad64-6646-4d3b-af99-53d142ed7a55-kube-api-access-k2n4j\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.341188 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.341151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf8ad64-6646-4d3b-af99-53d142ed7a55-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.356167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.356101    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n4j\" (UniqueName: \"kubernetes.io/projected/0cf8ad64-6646-4d3b-af99-53d142ed7a55-kube-api-access-k2n4j\") pod \"managed-serviceaccount-addon-agent-6995cbb6dd-htzv5\" (UID: \"0cf8ad64-6646-4d3b-af99-53d142ed7a55\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.442801 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.442450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"
Jul 29 10:29:59.622073 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.621981    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5"]
Jul 29 10:29:59.627739 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:29:59.627687    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cf8ad64_6646_4d3b_af99_53d142ed7a55.slice/crio-7422848e55b716efab3147e34f1e5db13906a4eb6c010b192e222804ce0915ec WatchSource:0}: Error finding container 7422848e55b716efab3147e34f1e5db13906a4eb6c010b192e222804ce0915ec: Status 404 returned error can't find the container with id 7422848e55b716efab3147e34f1e5db13906a4eb6c010b192e222804ce0915ec
Jul 29 10:29:59.761306 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.761147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6pkk" event={"ID":"5e233220-65a3-4899-b378-8fbccc025d1a","Type":"ContainerStarted","Data":"2e20b4702c97147c72846b0348e89349c803e415a4eaab4e89167793c7974ee6"}
Jul 29 10:29:59.762888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.762810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5" event={"ID":"0cf8ad64-6646-4d3b-af99-53d142ed7a55","Type":"ContainerStarted","Data":"7422848e55b716efab3147e34f1e5db13906a4eb6c010b192e222804ce0915ec"}
Jul 29 10:29:59.766155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.765345    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:29:59.766155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.765446    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" event={"ID":"20188e15-550f-46cd-b99b-a5bf59756d5b","Type":"ContainerStarted","Data":"936c7c4b3ec12017c80c88c214d9d2ac57a199ac35063b91b019adad1730bdb0"}
Jul 29 10:29:59.766446 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.766187    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:59.773105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.773079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx"
Jul 29 10:29:59.778338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.778310    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-6kv2b"]
Jul 29 10:29:59.783036 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.783013    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:29:59.786921 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.786686    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 10:29:59.786921 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.786754    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hfqfd\""
Jul 29 10:29:59.786921 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.786880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 10:29:59.787990 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.787939    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-bmxdx" podStartSLOduration=23.433972455 podStartE2EDuration="27.787924348s" totalImagesPullingTime="4.353951893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:29:59.786934436 +0000 UTC m=+75.967316327" watchObservedRunningTime="2026-07-29 10:29:59.787924348 +0000 UTC m=+75.968306227"
Jul 29 10:29:59.791152 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.791127    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6kv2b"]
Jul 29 10:29:59.845997 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.845964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9nzg\" (UniqueName: \"kubernetes.io/projected/52732dce-c95d-4b8a-b2f2-463b0102037b-kube-api-access-l9nzg\") pod \"downloads-7c97dc4555-6kv2b\" (UID: \"52732dce-c95d-4b8a-b2f2-463b0102037b\") " pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:29:59.947046 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.946964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9nzg\" (UniqueName: \"kubernetes.io/projected/52732dce-c95d-4b8a-b2f2-463b0102037b-kube-api-access-l9nzg\") pod \"downloads-7c97dc4555-6kv2b\" (UID: \"52732dce-c95d-4b8a-b2f2-463b0102037b\") " pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:29:59.972906 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:29:59.972861    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9nzg\" (UniqueName: \"kubernetes.io/projected/52732dce-c95d-4b8a-b2f2-463b0102037b-kube-api-access-l9nzg\") pod \"downloads-7c97dc4555-6kv2b\" (UID: \"52732dce-c95d-4b8a-b2f2-463b0102037b\") " pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:30:00.099134 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:00.099101    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:30:00.253809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:00.253774    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6kv2b"]
Jul 29 10:30:00.259387 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:00.259356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52732dce_c95d_4b8a_b2f2_463b0102037b.slice/crio-52ef984d038a41e91fba124ad491b57c7c800ed27c38969c7dedbd999a5acd2c WatchSource:0}: Error finding container 52ef984d038a41e91fba124ad491b57c7c800ed27c38969c7dedbd999a5acd2c: Status 404 returned error can't find the container with id 52ef984d038a41e91fba124ad491b57c7c800ed27c38969c7dedbd999a5acd2c
Jul 29 10:30:00.770934 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:00.770902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6kv2b" event={"ID":"52732dce-c95d-4b8a-b2f2-463b0102037b","Type":"ContainerStarted","Data":"52ef984d038a41e91fba124ad491b57c7c800ed27c38969c7dedbd999a5acd2c"}
Jul 29 10:30:02.780952 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:02.780881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6pkk" event={"ID":"5e233220-65a3-4899-b378-8fbccc025d1a","Type":"ContainerStarted","Data":"3098b242b2c4b901c6154d9c3aa2bbae1fda01ce97794215499f1221aa4dcfc4"}
Jul 29 10:30:04.690924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:04.690875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:30:04.700175 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:04.700134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/825761a1-4047-4058-b59b-24871c8e45e5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-cdrjg\" (UID: \"825761a1-4047-4058-b59b-24871c8e45e5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:30:04.886818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:04.886781    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-wm6pt\""
Jul 29 10:30:04.895799 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:04.895763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"
Jul 29 10:30:05.741304 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:05.740748    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z2cth"
Jul 29 10:30:07.302969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.302934    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:07.479990 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.479953    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:07.480179 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.480047    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.482576 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.482334    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 10:30:07.482576 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.482393    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 10:30:07.482576 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.482409    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 10:30:07.482833 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.482796    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 10:30:07.482893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.482841    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-k4brd\""
Jul 29 10:30:07.483199 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.483178    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 10:30:07.616831 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616750    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.616831 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.617028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.617028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.617028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616901    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.617028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.616929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs644\" (UniqueName: \"kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.717925 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.717890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.717964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.717995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.718027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.718066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs644\" (UniqueName: \"kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718397 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.718113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.718947 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.718889    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.719072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.719044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.719378 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.719351    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.721293 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.721270    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.721434 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.721367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.727475 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.727278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs644\" (UniqueName: \"kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644\") pod \"console-59d4f7f9fb-5gfwp\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") " pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:07.794231 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:07.794176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:08.310787 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.310741    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:08.315184 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:08.315143    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166b2149_e7dd_426c_a1de_12cb2b5e5113.slice/crio-1050311b6763fecfe83d94251adbdc7678a553a060e04182c23a2b1f1066fda7 WatchSource:0}: Error finding container 1050311b6763fecfe83d94251adbdc7678a553a060e04182c23a2b1f1066fda7: Status 404 returned error can't find the container with id 1050311b6763fecfe83d94251adbdc7678a553a060e04182c23a2b1f1066fda7
Jul 29 10:30:08.328776 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.328746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg"]
Jul 29 10:30:08.676028 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.675997    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"]
Jul 29 10:30:08.678679 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.678650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:08.680697 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.680673    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8dt4d\""
Jul 29 10:30:08.680697 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.680702    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 10:30:08.686856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.686206    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"]
Jul 29 10:30:08.806494 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.806465    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg" event={"ID":"825761a1-4047-4058-b59b-24871c8e45e5","Type":"ContainerStarted","Data":"5ceff06a20b6b424266300f4f2102dd37bae0a82f5b80b3b22a1e246c729932f"}
Jul 29 10:30:08.809001 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.808969    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q6pkk" event={"ID":"5e233220-65a3-4899-b378-8fbccc025d1a","Type":"ContainerStarted","Data":"ef77a2e73ff3d771b100be66e90af385bb4c935c26d921cfa1efe2519da6c6e5"}
Jul 29 10:30:08.810690 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.810663    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5" event={"ID":"0cf8ad64-6646-4d3b-af99-53d142ed7a55","Type":"ContainerStarted","Data":"2c44a049b52e1001df1394e267bc7627021d81708c1596aefc8716a1b8a61d80"}
Jul 29 10:30:08.811915 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.811890    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4f7f9fb-5gfwp" event={"ID":"166b2149-e7dd-426c-a1de-12cb2b5e5113","Type":"ContainerStarted","Data":"1050311b6763fecfe83d94251adbdc7678a553a060e04182c23a2b1f1066fda7"}
Jul 29 10:30:08.826555 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.826524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2dqsh\" (UID: \"916a79bd-ceb2-437c-bae4-03eaa853c62f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:08.828461 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.828399    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-q6pkk" podStartSLOduration=18.370380014 podStartE2EDuration="32.828378975s" totalImagesPullingTime="14.457998961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 10:29:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:08.827168694 +0000 UTC m=+85.007550575" watchObservedRunningTime="2026-07-29 10:30:08.828378975 +0000 UTC m=+85.008760857"
Jul 29 10:30:08.844311 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.844240    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6995cbb6dd-htzv5" podStartSLOduration=1.3615212190000001 podStartE2EDuration="9.844197768s" totalImagesPullingTime="8.482676549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:08.843044137 +0000 UTC m=+85.023426033" watchObservedRunningTime="2026-07-29 10:30:08.844197768 +0000 UTC m=+85.024579652"
Jul 29 10:30:08.928275 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:08.927786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2dqsh\" (UID: \"916a79bd-ceb2-437c-bae4-03eaa853c62f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:08.928472 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:30:08.928349    2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:30:08.928472 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:30:08.928440    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates podName:916a79bd-ceb2-437c-bae4-03eaa853c62f nodeName:}" failed. No retries permitted until 2026-07-29 10:30:09.428419065 +0000 UTC m=+85.608800940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-2dqsh" (UID: "916a79bd-ceb2-437c-bae4-03eaa853c62f") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 29 10:30:09.432561 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:09.432492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2dqsh\" (UID: \"916a79bd-ceb2-437c-bae4-03eaa853c62f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:09.435389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:09.435359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/916a79bd-ceb2-437c-bae4-03eaa853c62f-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-2dqsh\" (UID: \"916a79bd-ceb2-437c-bae4-03eaa853c62f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:09.590342 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:09.590073    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:09.763995 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:09.763957    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"]
Jul 29 10:30:09.768753 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:09.768714    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916a79bd_ceb2_437c_bae4_03eaa853c62f.slice/crio-bc7d7692e8840deee3296cb11d5be6fe8ac4bd8b4d372258781f0f1d069ce202 WatchSource:0}: Error finding container bc7d7692e8840deee3296cb11d5be6fe8ac4bd8b4d372258781f0f1d069ce202: Status 404 returned error can't find the container with id bc7d7692e8840deee3296cb11d5be6fe8ac4bd8b4d372258781f0f1d069ce202
Jul 29 10:30:09.823038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:09.822995    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh" event={"ID":"916a79bd-ceb2-437c-bae4-03eaa853c62f","Type":"ContainerStarted","Data":"bc7d7692e8840deee3296cb11d5be6fe8ac4bd8b4d372258781f0f1d069ce202"}
Jul 29 10:30:10.830115 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:10.830075    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg" event={"ID":"825761a1-4047-4058-b59b-24871c8e45e5","Type":"ContainerStarted","Data":"59cd94417aa50290bdef7a8d825de9020c46dd0e95cd5e268f5d8a03accf947d"}
Jul 29 10:30:10.844851 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:10.844782    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-cdrjg" podStartSLOduration=36.855324873 podStartE2EDuration="38.844765816s" totalImagesPullingTime="1.989440943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:10.843640104 +0000 UTC m=+87.024021985" watchObservedRunningTime="2026-07-29 10:30:10.844765816 +0000 UTC m=+87.025147697"
Jul 29 10:30:11.811580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:11.811543    2574 patch_prober.go:28] interesting pod/image-registry-65bb9b8556-g49sd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 10:30:11.811580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:11.811610    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 10:30:12.840957 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.840919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4f7f9fb-5gfwp" event={"ID":"166b2149-e7dd-426c-a1de-12cb2b5e5113","Type":"ContainerStarted","Data":"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"}
Jul 29 10:30:12.843133 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.843100    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh" event={"ID":"916a79bd-ceb2-437c-bae4-03eaa853c62f","Type":"ContainerStarted","Data":"0d80860e849ba4ba46998d0900dbd9e23f3bd5205c4653397eea1bb6fb0d87e6"}
Jul 29 10:30:12.843376 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.843352    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:12.848905 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.848879    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh"
Jul 29 10:30:12.857770 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.857714    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d4f7f9fb-5gfwp" podStartSLOduration=1.9799302779999999 podStartE2EDuration="5.857689299s" totalImagesPullingTime="3.877759021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:12.856642911 +0000 UTC m=+89.037024791" watchObservedRunningTime="2026-07-29 10:30:12.857689299 +0000 UTC m=+89.038071160"
Jul 29 10:30:12.871874 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:12.871810    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-2dqsh" podStartSLOduration=2.449951238 podStartE2EDuration="4.871788964s" totalImagesPullingTime="2.421837726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:12.870450087 +0000 UTC m=+89.050831972" watchObservedRunningTime="2026-07-29 10:30:12.871788964 +0000 UTC m=+89.052170844"
Jul 29 10:30:13.742141 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.742102    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-flfln"]
Jul 29 10:30:13.746748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.746724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.749689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.749665    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 10:30:13.749850 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.749716    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 10:30:13.749942 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.749924    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 29 10:30:13.749992 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.749939    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 29 10:30:13.750038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.749666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-86fxg\""
Jul 29 10:30:13.750262 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.750240    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 10:30:13.755498 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.754948    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-flfln"]
Jul 29 10:30:13.873609 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.873569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvlq\" (UniqueName: \"kubernetes.io/projected/f855927d-a2bb-4ebf-b294-a7ffb58788d5-kube-api-access-7kvlq\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.874016 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.873646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f855927d-a2bb-4ebf-b294-a7ffb58788d5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.874016 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.873734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.874016 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.873838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.975288 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.975251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f855927d-a2bb-4ebf-b294-a7ffb58788d5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.975485 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.975326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.975485 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.975406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.975485 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.975475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvlq\" (UniqueName: \"kubernetes.io/projected/f855927d-a2bb-4ebf-b294-a7ffb58788d5-kube-api-access-7kvlq\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.976095 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.976067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f855927d-a2bb-4ebf-b294-a7ffb58788d5-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.978580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.978550    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.978881 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.978847    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f855927d-a2bb-4ebf-b294-a7ffb58788d5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:13.984740 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:13.984711    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvlq\" (UniqueName: \"kubernetes.io/projected/f855927d-a2bb-4ebf-b294-a7ffb58788d5-kube-api-access-7kvlq\") pod \"prometheus-operator-84d69f794d-flfln\" (UID: \"f855927d-a2bb-4ebf-b294-a7ffb58788d5\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:14.058526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:14.058486    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln"
Jul 29 10:30:14.737367 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:14.737340    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:30:17.794770 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:17.794737    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:17.795349 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:17.794809    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:17.800426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:17.800400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:17.871515 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:17.871483    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:19.829876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:19.829840    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-flfln"]
Jul 29 10:30:19.992740 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:19.992661    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf855927d_a2bb_4ebf_b294_a7ffb58788d5.slice/crio-c97b919162e4dadb3907788657d58ae367f38dd881f8fbf31c3b15ba444b36fc WatchSource:0}: Error finding container c97b919162e4dadb3907788657d58ae367f38dd881f8fbf31c3b15ba444b36fc: Status 404 returned error can't find the container with id c97b919162e4dadb3907788657d58ae367f38dd881f8fbf31c3b15ba444b36fc
Jul 29 10:30:20.707731 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.707645    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrh9c"
Jul 29 10:30:20.884263 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.883827    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6kv2b" event={"ID":"52732dce-c95d-4b8a-b2f2-463b0102037b","Type":"ContainerStarted","Data":"7adf341c9c92f9534ccfbf78809b2cf9b421459456a094e08cd23f86d2de2492"}
Jul 29 10:30:20.884263 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.884144    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:30:20.886871 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.886829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln" event={"ID":"f855927d-a2bb-4ebf-b294-a7ffb58788d5","Type":"ContainerStarted","Data":"c97b919162e4dadb3907788657d58ae367f38dd881f8fbf31c3b15ba444b36fc"}
Jul 29 10:30:20.900033 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.899972    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-6kv2b"
Jul 29 10:30:20.900798 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:20.900743    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-6kv2b" podStartSLOduration=1.819490737 podStartE2EDuration="21.90072475s" totalImagesPullingTime="20.081234013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:20.900025006 +0000 UTC m=+97.080406889" watchObservedRunningTime="2026-07-29 10:30:20.90072475 +0000 UTC m=+97.081106631"
Jul 29 10:30:22.458804 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:22.458766    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:30:22.897395 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:22.897345    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln" event={"ID":"f855927d-a2bb-4ebf-b294-a7ffb58788d5","Type":"ContainerStarted","Data":"10ad478cbf15cae4999cd66647362d82cc93bd3d93cfa05a43b3c8c15bcbee04"}
Jul 29 10:30:22.897582 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:22.897419    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln" event={"ID":"f855927d-a2bb-4ebf-b294-a7ffb58788d5","Type":"ContainerStarted","Data":"e65b0d2d9161b74a21fb822eba386570fbc4cecf8174488bbbe3c6508836e6f4"}
Jul 29 10:30:22.916475 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:22.916413    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-flfln" podStartSLOduration=7.892169289 podStartE2EDuration="9.916396364s" totalImagesPullingTime="2.024227075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:22.915508391 +0000 UTC m=+99.095890271" watchObservedRunningTime="2026-07-29 10:30:22.916396364 +0000 UTC m=+99.096778244"
Jul 29 10:30:25.136251 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.136201    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rfbrk"]
Jul 29 10:30:25.179780 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.179740    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.182345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.182316    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 10:30:25.182483 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.182437    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nhtns\""
Jul 29 10:30:25.182671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.182656    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 10:30:25.182881 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.182864    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 10:30:25.288094 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-root\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288094 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-tls\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-sys\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9pn\" (UniqueName: \"kubernetes.io/projected/49ae2060-41df-42dd-ab80-7787fa700f8a-kube-api-access-hv9pn\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-wtmp\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288589 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-textfile\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288589 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-metrics-client-ca\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.288589 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.288473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389443 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-textfile\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389443 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-metrics-client-ca\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389443 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-root\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-root\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-tls\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-sys\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.389739 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389720    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9pn\" (UniqueName: \"kubernetes.io/projected/49ae2060-41df-42dd-ab80-7787fa700f8a-kube-api-access-hv9pn\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390034 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-wtmp\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390034 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389815    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-textfile\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390034 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389929    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-wtmp\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390034 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.389958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ae2060-41df-42dd-ab80-7787fa700f8a-sys\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390521 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.390495    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-metrics-client-ca\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.390651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.390591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-accelerators-collector-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.392488 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.392457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.392697 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.392680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ae2060-41df-42dd-ab80-7787fa700f8a-node-exporter-tls\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.398317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.398288    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9pn\" (UniqueName: \"kubernetes.io/projected/49ae2060-41df-42dd-ab80-7787fa700f8a-kube-api-access-hv9pn\") pod \"node-exporter-rfbrk\" (UID: \"49ae2060-41df-42dd-ab80-7787fa700f8a\") " pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.490629 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.490595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rfbrk"
Jul 29 10:30:25.531292 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:25.531256    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ae2060_41df_42dd_ab80_7787fa700f8a.slice/crio-c1b3cce6ad0555b8026235294f4c573dec59987abc93366e2dd379925f8f3ea0 WatchSource:0}: Error finding container c1b3cce6ad0555b8026235294f4c573dec59987abc93366e2dd379925f8f3ea0: Status 404 returned error can't find the container with id c1b3cce6ad0555b8026235294f4c573dec59987abc93366e2dd379925f8f3ea0
Jul 29 10:30:25.911239 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:25.911182    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbrk" event={"ID":"49ae2060-41df-42dd-ab80-7787fa700f8a","Type":"ContainerStarted","Data":"c1b3cce6ad0555b8026235294f4c573dec59987abc93366e2dd379925f8f3ea0"}
Jul 29 10:30:27.589369 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:27.589292    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:27.921927 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:27.921838    2574 generic.go:356] "Generic (PLEG): container finished" podID="49ae2060-41df-42dd-ab80-7787fa700f8a" containerID="052772c789c279a2a1edb5fe5293453269357de70bafb4b5d7679974789e2a1e" exitCode=0
Jul 29 10:30:27.921927 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:27.921891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbrk" event={"ID":"49ae2060-41df-42dd-ab80-7787fa700f8a","Type":"ContainerDied","Data":"052772c789c279a2a1edb5fe5293453269357de70bafb4b5d7679974789e2a1e"}
Jul 29 10:30:28.929084 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:28.929046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbrk" event={"ID":"49ae2060-41df-42dd-ab80-7787fa700f8a","Type":"ContainerStarted","Data":"9516adb742a0ef440ca74506f3b7a12ef82da7c6b75462e6ed6f8c89e5aee535"}
Jul 29 10:30:28.929624 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:28.929113    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rfbrk" event={"ID":"49ae2060-41df-42dd-ab80-7787fa700f8a","Type":"ContainerStarted","Data":"25c7a4c4f4594c88512370c21caa3f14f8b9c8cfb360cb9379caf96a9e23c894"}
Jul 29 10:30:28.950010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:28.949951    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rfbrk" podStartSLOduration=2.035584426 podStartE2EDuration="3.949933186s" totalImagesPullingTime="1.794585941s" totalInitContainerRuntime="119.762819ms" isStatefulPod=true podCreationTimestamp="2026-07-29 10:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:28.949332299 +0000 UTC m=+105.129714181" watchObservedRunningTime="2026-07-29 10:30:28.949933186 +0000 UTC m=+105.130315066"
Jul 29 10:30:29.912889 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:29.912849    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:30:29.957272 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:29.957200    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:30:29.957764 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:29.957333    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:29.967333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:29.967309    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 10:30:30.041855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.041809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.041855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.041862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jcz\" (UniqueName: \"kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.042120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.042027    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.042120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.042084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.042262 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.042189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.042464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.042443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.042570 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.042473    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.142980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143297 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143054    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143297 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143297 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.143431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.143409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jcz\" (UniqueName: \"kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.144029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.144001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.144153 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.144035    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.144153 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.144026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.144153 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.144055    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.146495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.146468    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.146605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.146539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.152230 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.152184    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jcz\" (UniqueName: \"kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz\") pod \"console-6c45484945-b6zd6\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") " pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.269860 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.269828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:30.438982 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.438946    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:30:30.442343 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:30.442310    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51058cef_1ee0_4aea_8d28_571dcfc71876.slice/crio-a33502a811d33e20c109e074101df63651e44eac0268f3aec52d66d58f43ed5b WatchSource:0}: Error finding container a33502a811d33e20c109e074101df63651e44eac0268f3aec52d66d58f43ed5b: Status 404 returned error can't find the container with id a33502a811d33e20c109e074101df63651e44eac0268f3aec52d66d58f43ed5b
Jul 29 10:30:30.939355 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.939254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45484945-b6zd6" event={"ID":"51058cef-1ee0-4aea-8d28-571dcfc71876","Type":"ContainerStarted","Data":"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"}
Jul 29 10:30:30.939355 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.939307    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45484945-b6zd6" event={"ID":"51058cef-1ee0-4aea-8d28-571dcfc71876","Type":"ContainerStarted","Data":"a33502a811d33e20c109e074101df63651e44eac0268f3aec52d66d58f43ed5b"}
Jul 29 10:30:30.959130 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:30.959069    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c45484945-b6zd6" podStartSLOduration=1.959051614 podStartE2EDuration="1.959051614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:30.957360264 +0000 UTC m=+107.137742145" watchObservedRunningTime="2026-07-29 10:30:30.959051614 +0000 UTC m=+107.139433495"
Jul 29 10:30:40.270898 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:40.270859    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:40.271549 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:40.270926    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:40.275131 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:40.275104    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:40.977410 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:40.977383    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:30:47.485348 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.485278    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerName="registry" containerID="cri-o://e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79" gracePeriod=30
Jul 29 10:30:47.772856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.772817    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:30:47.902025 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.901988    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh6ch\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902025 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902030    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902052    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902110    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902129    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902171    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902203    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets\") pod \"453a394d-7d32-490a-9ac2-3be63bb6d98d\" (UID: \"453a394d-7d32-490a-9ac2-3be63bb6d98d\") "
Jul 29 10:30:47.902621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902505    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:47.902771 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.902744    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:47.905021 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.904946    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:47.905021 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.904997    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:47.905301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.905052    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch" (OuterVolumeSpecName: "kube-api-access-qh6ch") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "kube-api-access-qh6ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:47.905301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.905251    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:47.905301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.905258    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:47.911020 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:47.910992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "453a394d-7d32-490a-9ac2-3be63bb6d98d" (UID: "453a394d-7d32-490a-9ac2-3be63bb6d98d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:30:48.002809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002716    2574 generic.go:356] "Generic (PLEG): container finished" podID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerID="e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79" exitCode=0
Jul 29 10:30:48.002809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002776    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd"
Jul 29 10:30:48.002809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002791    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" event={"ID":"453a394d-7d32-490a-9ac2-3be63bb6d98d","Type":"ContainerDied","Data":"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"}
Jul 29 10:30:48.002809 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002805    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-trusted-ca\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002826    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-image-registry-private-configuration\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65bb9b8556-g49sd" event={"ID":"453a394d-7d32-490a-9ac2-3be63bb6d98d","Type":"ContainerDied","Data":"76cd37fe9ee3f8c517fd262af62b00e4786a63bcbc82ae1a2cbb7efc84405dd0"}
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002839    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-tls\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002851    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-bound-sa-token\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002862    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/453a394d-7d32-490a-9ac2-3be63bb6d98d-registry-certificates\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002875    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/453a394d-7d32-490a-9ac2-3be63bb6d98d-ca-trust-extracted\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002887    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/453a394d-7d32-490a-9ac2-3be63bb6d98d-installation-pull-secrets\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002852    2574 scope.go:117] "RemoveContainer" containerID="e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"
Jul 29 10:30:48.003105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.002932    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh6ch\" (UniqueName: \"kubernetes.io/projected/453a394d-7d32-490a-9ac2-3be63bb6d98d-kube-api-access-qh6ch\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:48.016825 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.016804    2574 scope.go:117] "RemoveContainer" containerID="e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"
Jul 29 10:30:48.017157 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:30:48.017137    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79\": container with ID starting with e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79 not found: ID does not exist" containerID="e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"
Jul 29 10:30:48.017209 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.017168    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79"} err="failed to get container status \"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79\": rpc error: code = NotFound desc = could not find container \"e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79\": container with ID starting with e1e52d0bcd3bbe77a45fe880911659ecb611447bd30e3d29ae9f0a9241563b79 not found: ID does not exist"
Jul 29 10:30:48.022710 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.022686    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:30:48.026053 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.026031    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65bb9b8556-g49sd"]
Jul 29 10:30:48.340947 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:48.340914    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" path="/var/lib/kubelet/pods/453a394d-7d32-490a-9ac2-3be63bb6d98d/volumes"
Jul 29 10:30:52.617291 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:52.617246    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59d4f7f9fb-5gfwp" podUID="166b2149-e7dd-426c-a1de-12cb2b5e5113" containerName="console" containerID="cri-o://58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7" gracePeriod=15
Jul 29 10:30:52.892020 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:52.891947    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:53.021585 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.021557    2574 generic.go:356] "Generic (PLEG): container finished" podID="166b2149-e7dd-426c-a1de-12cb2b5e5113" containerID="58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7" exitCode=0
Jul 29 10:30:53.021773 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.021603    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4f7f9fb-5gfwp"
Jul 29 10:30:53.021773 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.021635    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4f7f9fb-5gfwp" event={"ID":"166b2149-e7dd-426c-a1de-12cb2b5e5113","Type":"ContainerDied","Data":"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"}
Jul 29 10:30:53.021773 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.021677    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4f7f9fb-5gfwp" event={"ID":"166b2149-e7dd-426c-a1de-12cb2b5e5113","Type":"ContainerDied","Data":"1050311b6763fecfe83d94251adbdc7678a553a060e04182c23a2b1f1066fda7"}
Jul 29 10:30:53.021773 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.021697    2574 scope.go:117] "RemoveContainer" containerID="58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"
Jul 29 10:30:53.029528 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.029509    2574 scope.go:117] "RemoveContainer" containerID="58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"
Jul 29 10:30:53.029793 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:30:53.029774    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7\": container with ID starting with 58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7 not found: ID does not exist" containerID="58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"
Jul 29 10:30:53.029846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.029803    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7"} err="failed to get container status \"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7\": rpc error: code = NotFound desc = could not find container \"58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7\": container with ID starting with 58aff8dad66a4ba62dd422597af6ece370c5c8f7dbc3b7bb3f372759fd076fe7 not found: ID does not exist"
Jul 29 10:30:53.044088 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs644\" (UniqueName: \"kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044103    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044129    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044151    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044363 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044194    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044363 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044272    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert\") pod \"166b2149-e7dd-426c-a1de-12cb2b5e5113\" (UID: \"166b2149-e7dd-426c-a1de-12cb2b5e5113\") "
Jul 29 10:30:53.044694 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config" (OuterVolumeSpecName: "console-config") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:53.044807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044700    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca" (OuterVolumeSpecName: "service-ca") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:53.044807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.044748    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:30:53.046458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.046435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:53.046550 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.046510    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644" (OuterVolumeSpecName: "kube-api-access-rs644") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "kube-api-access-rs644". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:30:53.046609 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.046594    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "166b2149-e7dd-426c-a1de-12cb2b5e5113" (UID: "166b2149-e7dd-426c-a1de-12cb2b5e5113"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145190    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-oauth-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145248    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs644\" (UniqueName: \"kubernetes.io/projected/166b2149-e7dd-426c-a1de-12cb2b5e5113-kube-api-access-rs644\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145258    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145269    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-oauth-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145278    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-console-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.145280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.145286    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/166b2149-e7dd-426c-a1de-12cb2b5e5113-service-ca\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:30:53.246013 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.245973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:30:53.248316 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.248290    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2df1e671-4dab-4eed-b983-326d3fc53b9c-metrics-certs\") pod \"network-metrics-daemon-5fm96\" (UID: \"2df1e671-4dab-4eed-b983-326d3fc53b9c\") " pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:30:53.341998 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.341967    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:53.346312 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.346282    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d4f7f9fb-5gfwp"]
Jul 29 10:30:53.448449 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.448369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kqgtv\""
Jul 29 10:30:53.456919 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.456888    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5fm96"
Jul 29 10:30:53.612157 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:53.612129    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5fm96"]
Jul 29 10:30:53.614638 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:30:53.614607    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df1e671_4dab_4eed_b983_326d3fc53b9c.slice/crio-ed278555d14728da030580d1204f2c0abe048fe6420a84dd83653f546828ef4d WatchSource:0}: Error finding container ed278555d14728da030580d1204f2c0abe048fe6420a84dd83653f546828ef4d: Status 404 returned error can't find the container with id ed278555d14728da030580d1204f2c0abe048fe6420a84dd83653f546828ef4d
Jul 29 10:30:54.027564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:54.027529    2574 generic.go:356] "Generic (PLEG): container finished" podID="95ec880c-ff89-4c96-91d7-a390b557c278" containerID="710ae889a7e6813fc69cf72ac98988f555100caf32f3a84461797d588107f3bf" exitCode=0
Jul 29 10:30:54.027974 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:54.027598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb" event={"ID":"95ec880c-ff89-4c96-91d7-a390b557c278","Type":"ContainerDied","Data":"710ae889a7e6813fc69cf72ac98988f555100caf32f3a84461797d588107f3bf"}
Jul 29 10:30:54.027974 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:54.027955    2574 scope.go:117] "RemoveContainer" containerID="710ae889a7e6813fc69cf72ac98988f555100caf32f3a84461797d588107f3bf"
Jul 29 10:30:54.028899 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:54.028875    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fm96" event={"ID":"2df1e671-4dab-4eed-b983-326d3fc53b9c","Type":"ContainerStarted","Data":"ed278555d14728da030580d1204f2c0abe048fe6420a84dd83653f546828ef4d"}
Jul 29 10:30:54.342415 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:54.342329    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="166b2149-e7dd-426c-a1de-12cb2b5e5113" path="/var/lib/kubelet/pods/166b2149-e7dd-426c-a1de-12cb2b5e5113/volumes"
Jul 29 10:30:55.037141 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:55.037098    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb" event={"ID":"95ec880c-ff89-4c96-91d7-a390b557c278","Type":"ContainerStarted","Data":"70c1cfc69ff6e14625545d5253e02043919246d00b6c8574143f21adac7d087c"}
Jul 29 10:30:56.042438 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:56.042404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fm96" event={"ID":"2df1e671-4dab-4eed-b983-326d3fc53b9c","Type":"ContainerStarted","Data":"48bef12d69505694a72eca9e94c2b83167648eb0d5ed0f04172f58b5509cb8dd"}
Jul 29 10:30:56.042838 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:56.042464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5fm96" event={"ID":"2df1e671-4dab-4eed-b983-326d3fc53b9c","Type":"ContainerStarted","Data":"5bfe11ebcbc6ddf3bc8892b5d24cf46cd568b10bed0dc97ce9696850ad10223b"}
Jul 29 10:30:56.058181 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:30:56.058128    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5fm96" podStartSLOduration=129.476691794 podStartE2EDuration="2m11.058113917s" totalImagesPullingTime="1.581422123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:30:56.057240758 +0000 UTC m=+132.237622633" watchObservedRunningTime="2026-07-29 10:30:56.058113917 +0000 UTC m=+132.238495850"
Jul 29 10:31:49.348754 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.348716    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"]
Jul 29 10:31:49.349708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349679    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="166b2149-e7dd-426c-a1de-12cb2b5e5113" containerName="console"
Jul 29 10:31:49.349708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349707    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="166b2149-e7dd-426c-a1de-12cb2b5e5113" containerName="console"
Jul 29 10:31:49.349908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349723    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerName="registry"
Jul 29 10:31:49.349908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349734    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerName="registry"
Jul 29 10:31:49.349908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349792    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="453a394d-7d32-490a-9ac2-3be63bb6d98d" containerName="registry"
Jul 29 10:31:49.349908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.349801    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="166b2149-e7dd-426c-a1de-12cb2b5e5113" containerName="console"
Jul 29 10:31:49.352732 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.352709    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.355329 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.355304    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 29 10:31:49.355600 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.355573    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 29 10:31:49.355728 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.355594    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 29 10:31:49.355728 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.355637    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 29 10:31:49.355952 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.355923    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 29 10:31:49.356027 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.356009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6t6f4\""
Jul 29 10:31:49.362270 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.362211    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 29 10:31:49.363878 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.363853    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"]
Jul 29 10:31:49.481032 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.480982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481032 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481161    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh4b2\" (UniqueName: \"kubernetes.io/projected/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-kube-api-access-wh4b2\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-metrics-client-ca\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481558 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-federate-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.481558 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.481450    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-serving-certs-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh4b2\" (UniqueName: \"kubernetes.io/projected/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-kube-api-access-wh4b2\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-metrics-client-ca\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-federate-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582778    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-serving-certs-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.582971 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.582878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.583681 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.583622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-metrics-client-ca\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.583816 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.583691    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-serving-certs-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.584195 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.584166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.585464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.585443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.585677 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.585656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-telemeter-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.585677 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.585667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-federate-client-tls\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.585748 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.585729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.590648 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.590625    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh4b2\" (UniqueName: \"kubernetes.io/projected/795c50eb-aa9a-46eb-9c7b-b8efe347a2ec-kube-api-access-wh4b2\") pod \"telemeter-client-5f995bbb7b-q4trf\" (UID: \"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec\") " pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.665850 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.665763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"
Jul 29 10:31:49.816326 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:49.816287    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f995bbb7b-q4trf"]
Jul 29 10:31:49.822538 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:31:49.822503    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795c50eb_aa9a_46eb_9c7b_b8efe347a2ec.slice/crio-af51c938d30acd2f2010ca84c9aca131b6e8e16ae6d691eb10e3ca69cde89cdb WatchSource:0}: Error finding container af51c938d30acd2f2010ca84c9aca131b6e8e16ae6d691eb10e3ca69cde89cdb: Status 404 returned error can't find the container with id af51c938d30acd2f2010ca84c9aca131b6e8e16ae6d691eb10e3ca69cde89cdb
Jul 29 10:31:50.243577 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:50.243537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf" event={"ID":"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec","Type":"ContainerStarted","Data":"af51c938d30acd2f2010ca84c9aca131b6e8e16ae6d691eb10e3ca69cde89cdb"}
Jul 29 10:31:52.253506 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:52.253463    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf" event={"ID":"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec","Type":"ContainerStarted","Data":"b61d04c7c8c57e5f46940971d1f281b42d0ad5ddb7df7c42548bf9f7c6c47332"}
Jul 29 10:31:53.259869 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:53.259832    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf" event={"ID":"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec","Type":"ContainerStarted","Data":"b87b54a619af2ae83fbb123f0de163100d944a25309a246f5da8ee44ab9f7b51"}
Jul 29 10:31:53.260286 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:53.259887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf" event={"ID":"795c50eb-aa9a-46eb-9c7b-b8efe347a2ec","Type":"ContainerStarted","Data":"2f932095e991e386940c6ba18d2d0c6e39dd4315fd4fac2687bdc4c98fc50de0"}
Jul 29 10:31:53.280456 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:53.280333    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f995bbb7b-q4trf" podStartSLOduration=1.6080306100000001 podStartE2EDuration="4.28031089s" totalImagesPullingTime="2.67228028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:31:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:31:53.279843354 +0000 UTC m=+189.460225233" watchObservedRunningTime="2026-07-29 10:31:53.28031089 +0000 UTC m=+189.460692771"
Jul 29 10:31:54.012245 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.012186    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:31:54.014863 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.014835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.027146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.027106    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:31:54.126035 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.125996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126035 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgpr\" (UniqueName: \"kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.126336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.126298    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.226843 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.226823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgpr\" (UniqueName: \"kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.227584 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.227551    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.227584 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.227560    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.227905 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.227843    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.228057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.228035    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.229431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.229400    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.229515 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.229478    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.236680 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.236662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgpr\" (UniqueName: \"kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr\") pod \"console-6946fcf5f4-8kmw8\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") " pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.324235 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.324193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:31:54.457305 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:54.457277    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:31:54.460382 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:31:54.460360    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22f3b6d_508c_442d_b530_cefc9aa75543.slice/crio-40a7c50179fd22c905b32875bec7d967c264e9b266b3b1eaf8864ce2f131cdde WatchSource:0}: Error finding container 40a7c50179fd22c905b32875bec7d967c264e9b266b3b1eaf8864ce2f131cdde: Status 404 returned error can't find the container with id 40a7c50179fd22c905b32875bec7d967c264e9b266b3b1eaf8864ce2f131cdde
Jul 29 10:31:55.268325 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:55.268285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6946fcf5f4-8kmw8" event={"ID":"c22f3b6d-508c-442d-b530-cefc9aa75543","Type":"ContainerStarted","Data":"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"}
Jul 29 10:31:55.268472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:55.268333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6946fcf5f4-8kmw8" event={"ID":"c22f3b6d-508c-442d-b530-cefc9aa75543","Type":"ContainerStarted","Data":"40a7c50179fd22c905b32875bec7d967c264e9b266b3b1eaf8864ce2f131cdde"}
Jul 29 10:31:55.287699 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:55.287633    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6946fcf5f4-8kmw8" podStartSLOduration=2.28761104 podStartE2EDuration="2.28761104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:31:55.286453156 +0000 UTC m=+191.466835037" watchObservedRunningTime="2026-07-29 10:31:55.28761104 +0000 UTC m=+191.467992922"
Jul 29 10:31:59.792648 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.792616    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:31:59.822911 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.822875    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b9cc94695-plmjd"]
Jul 29 10:31:59.825564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.825537    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.837398 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.837374    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cc94695-plmjd"]
Jul 29 10:31:59.977782 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-oauth-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.977959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-trusted-ca-bundle\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.977959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-oauth-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.977959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tfh\" (UniqueName: \"kubernetes.io/projected/815906ea-4f97-48bd-a50d-0aceee338f9e-kube-api-access-69tfh\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.977959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.977959 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-console-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:31:59.978119 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:31:59.977985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-service-ca\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-oauth-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-trusted-ca-bundle\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079148    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-oauth-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079395 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tfh\" (UniqueName: \"kubernetes.io/projected/815906ea-4f97-48bd-a50d-0aceee338f9e-kube-api-access-69tfh\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079395 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079395 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-console-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079395 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-service-ca\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.079987 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-oauth-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.080090 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.079989    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-service-ca\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.080090 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.080071    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-trusted-ca-bundle\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.080333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.080310    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/815906ea-4f97-48bd-a50d-0aceee338f9e-console-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.081839 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.081812    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-serving-cert\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.081920 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.081851    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/815906ea-4f97-48bd-a50d-0aceee338f9e-console-oauth-config\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.087485 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.087452    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tfh\" (UniqueName: \"kubernetes.io/projected/815906ea-4f97-48bd-a50d-0aceee338f9e-kube-api-access-69tfh\") pod \"console-b9cc94695-plmjd\" (UID: \"815906ea-4f97-48bd-a50d-0aceee338f9e\") " pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.134646 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.134608    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:00.269409 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.269369    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b9cc94695-plmjd"]
Jul 29 10:32:00.272956 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:32:00.272924    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815906ea_4f97_48bd_a50d_0aceee338f9e.slice/crio-490d4679eacfc291fd5c937dba0c1e175dbdf8d02f5b0fffc360c3689e1b8153 WatchSource:0}: Error finding container 490d4679eacfc291fd5c937dba0c1e175dbdf8d02f5b0fffc360c3689e1b8153: Status 404 returned error can't find the container with id 490d4679eacfc291fd5c937dba0c1e175dbdf8d02f5b0fffc360c3689e1b8153
Jul 29 10:32:00.288292 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:00.288262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cc94695-plmjd" event={"ID":"815906ea-4f97-48bd-a50d-0aceee338f9e","Type":"ContainerStarted","Data":"490d4679eacfc291fd5c937dba0c1e175dbdf8d02f5b0fffc360c3689e1b8153"}
Jul 29 10:32:01.293530 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:01.293492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b9cc94695-plmjd" event={"ID":"815906ea-4f97-48bd-a50d-0aceee338f9e","Type":"ContainerStarted","Data":"b4b810433df4a6a6dc7b5d2973d869368475a81868f598ca02117f047d737085"}
Jul 29 10:32:04.324354 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:04.324323    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:32:10.135537 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.135493    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:10.136009 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.135569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:10.140472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.140448    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:10.158149 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.158097    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b9cc94695-plmjd" podStartSLOduration=11.158080367 podStartE2EDuration="11.158080367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:32:01.332269347 +0000 UTC m=+197.512651252" watchObservedRunningTime="2026-07-29 10:32:10.158080367 +0000 UTC m=+206.338462246"
Jul 29 10:32:10.331473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.331446    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b9cc94695-plmjd"
Jul 29 10:32:10.376630 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:10.376594    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:32:24.812120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:24.812079    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6946fcf5f4-8kmw8" podUID="c22f3b6d-508c-442d-b530-cefc9aa75543" containerName="console" containerID="cri-o://4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116" gracePeriod=15
Jul 29 10:32:25.055614 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.055589    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:32:25.084743 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084658    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.084743 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084689    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.084743 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084714    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.085008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084747    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slgpr\" (UniqueName: \"kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.085008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084768    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.085008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084815    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.085008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.084862    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle\") pod \"c22f3b6d-508c-442d-b530-cefc9aa75543\" (UID: \"c22f3b6d-508c-442d-b530-cefc9aa75543\") "
Jul 29 10:32:25.085198 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.085164    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca" (OuterVolumeSpecName: "service-ca") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:25.085446 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.085409    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:25.085536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.085426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config" (OuterVolumeSpecName: "console-config") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:25.085730 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.085699    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:25.087114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.087087    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr" (OuterVolumeSpecName: "kube-api-access-slgpr") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "kube-api-access-slgpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:32:25.087114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.087093    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:25.087480 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.087462    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c22f3b6d-508c-442d-b530-cefc9aa75543" (UID: "c22f3b6d-508c-442d-b530-cefc9aa75543"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:25.185621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185580    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgpr\" (UniqueName: \"kubernetes.io/projected/c22f3b6d-508c-442d-b530-cefc9aa75543-kube-api-access-slgpr\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185613    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-oauth-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185623    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-console-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185631    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-trusted-ca-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185885 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185640    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185885 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185650    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c22f3b6d-508c-442d-b530-cefc9aa75543-console-oauth-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.185885 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.185657    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c22f3b6d-508c-442d-b530-cefc9aa75543-service-ca\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:25.382123 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.382028    2574 generic.go:356] "Generic (PLEG): container finished" podID="c22f3b6d-508c-442d-b530-cefc9aa75543" containerID="4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116" exitCode=0
Jul 29 10:32:25.382123 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.382080    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6946fcf5f4-8kmw8"
Jul 29 10:32:25.382359 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.382080    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6946fcf5f4-8kmw8" event={"ID":"c22f3b6d-508c-442d-b530-cefc9aa75543","Type":"ContainerDied","Data":"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"}
Jul 29 10:32:25.382359 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.382186    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6946fcf5f4-8kmw8" event={"ID":"c22f3b6d-508c-442d-b530-cefc9aa75543","Type":"ContainerDied","Data":"40a7c50179fd22c905b32875bec7d967c264e9b266b3b1eaf8864ce2f131cdde"}
Jul 29 10:32:25.382359 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.382212    2574 scope.go:117] "RemoveContainer" containerID="4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"
Jul 29 10:32:25.391152 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.391135    2574 scope.go:117] "RemoveContainer" containerID="4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"
Jul 29 10:32:25.391482 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:32:25.391459    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116\": container with ID starting with 4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116 not found: ID does not exist" containerID="4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"
Jul 29 10:32:25.391575 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.391488    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116"} err="failed to get container status \"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116\": rpc error: code = NotFound desc = could not find container \"4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116\": container with ID starting with 4ac102c286b9569413e7f8ebc16ca71c77bc1068a98e5aa5b5a30c2bf48ba116 not found: ID does not exist"
Jul 29 10:32:25.405866 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.404442    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:32:25.408590 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:25.408560    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6946fcf5f4-8kmw8"]
Jul 29 10:32:26.341235 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:26.341189    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c22f3b6d-508c-442d-b530-cefc9aa75543" path="/var/lib/kubelet/pods/c22f3b6d-508c-442d-b530-cefc9aa75543/volumes"
Jul 29 10:32:35.397108 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.397039    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c45484945-b6zd6" podUID="51058cef-1ee0-4aea-8d28-571dcfc71876" containerName="console" containerID="cri-o://c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce" gracePeriod=15
Jul 29 10:32:35.643894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.643864    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:32:35.770745 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.770704    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.770948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.770752    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9jcz\" (UniqueName: \"kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.770948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.770771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.770948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.770790    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.770948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.770935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.771112 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771027    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.771112 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771058    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config\") pod \"51058cef-1ee0-4aea-8d28-571dcfc71876\" (UID: \"51058cef-1ee0-4aea-8d28-571dcfc71876\") "
Jul 29 10:32:35.771309 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771278    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:35.771309 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771289    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config" (OuterVolumeSpecName: "console-config") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:35.771309 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771302    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:35.771556 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.771364    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca" (OuterVolumeSpecName: "service-ca") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 10:32:35.773306 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.773282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:35.773398 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.773314    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:32:35.773516 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.773497    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz" (OuterVolumeSpecName: "kube-api-access-c9jcz") pod "51058cef-1ee0-4aea-8d28-571dcfc71876" (UID: "51058cef-1ee0-4aea-8d28-571dcfc71876"). InnerVolumeSpecName "kube-api-access-c9jcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:32:35.872067 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872026    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-service-ca\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872067 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872060    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872067 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872075    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51058cef-1ee0-4aea-8d28-571dcfc71876-console-oauth-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872086    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-console-config\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872098    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jcz\" (UniqueName: \"kubernetes.io/projected/51058cef-1ee0-4aea-8d28-571dcfc71876-kube-api-access-c9jcz\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872109    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-trusted-ca-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:35.872338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:35.872120    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51058cef-1ee0-4aea-8d28-571dcfc71876-oauth-serving-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:32:36.424687 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.424650    2574 generic.go:356] "Generic (PLEG): container finished" podID="51058cef-1ee0-4aea-8d28-571dcfc71876" containerID="c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce" exitCode=0
Jul 29 10:32:36.425143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.424702    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c45484945-b6zd6"
Jul 29 10:32:36.425143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.424724    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45484945-b6zd6" event={"ID":"51058cef-1ee0-4aea-8d28-571dcfc71876","Type":"ContainerDied","Data":"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"}
Jul 29 10:32:36.425143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.424778    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c45484945-b6zd6" event={"ID":"51058cef-1ee0-4aea-8d28-571dcfc71876","Type":"ContainerDied","Data":"a33502a811d33e20c109e074101df63651e44eac0268f3aec52d66d58f43ed5b"}
Jul 29 10:32:36.425143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.424806    2574 scope.go:117] "RemoveContainer" containerID="c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"
Jul 29 10:32:36.433299 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.433279    2574 scope.go:117] "RemoveContainer" containerID="c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"
Jul 29 10:32:36.433589 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:32:36.433568    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce\": container with ID starting with c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce not found: ID does not exist" containerID="c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"
Jul 29 10:32:36.433656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.433597    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce"} err="failed to get container status \"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce\": rpc error: code = NotFound desc = could not find container \"c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce\": container with ID starting with c97201158881f0dfc9e2d4713ec76a5c6a7b070743c45c6e13e35bd437a532ce not found: ID does not exist"
Jul 29 10:32:36.438632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.438609    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:32:36.442143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:36.442118    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c45484945-b6zd6"]
Jul 29 10:32:38.340444 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:32:38.340412    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51058cef-1ee0-4aea-8d28-571dcfc71876" path="/var/lib/kubelet/pods/51058cef-1ee0-4aea-8d28-571dcfc71876/volumes"
Jul 29 10:33:28.772033 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.771986    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"]
Jul 29 10:33:28.772495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772479    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c22f3b6d-508c-442d-b530-cefc9aa75543" containerName="console"
Jul 29 10:33:28.772540 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772496    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c22f3b6d-508c-442d-b530-cefc9aa75543" containerName="console"
Jul 29 10:33:28.772540 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772514    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51058cef-1ee0-4aea-8d28-571dcfc71876" containerName="console"
Jul 29 10:33:28.772540 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772523    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51058cef-1ee0-4aea-8d28-571dcfc71876" containerName="console"
Jul 29 10:33:28.772632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772586    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c22f3b6d-508c-442d-b530-cefc9aa75543" containerName="console"
Jul 29 10:33:28.772632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.772598    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51058cef-1ee0-4aea-8d28-571dcfc71876" containerName="console"
Jul 29 10:33:28.776124 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.776100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.778414 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.778383    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 10:33:28.778563 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.778461    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 10:33:28.778836 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.778817    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhmdt\""
Jul 29 10:33:28.783845 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.783821    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"]
Jul 29 10:33:28.803616 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.803580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjl49\" (UniqueName: \"kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.803796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.803685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.803796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.803722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.904593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.904555    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.904803 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.904617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjl49\" (UniqueName: \"kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.904803 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.904655    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.905029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.905012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.905029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.905012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:28.912789 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:28.912757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjl49\" (UniqueName: \"kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:29.086569 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:29.086472    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:29.219431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:29.219398    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"]
Jul 29 10:33:29.222759 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:33:29.222730    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff411c05_36d3_41db_b8d7_a49afede66eb.slice/crio-ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0 WatchSource:0}: Error finding container ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0: Status 404 returned error can't find the container with id ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0
Jul 29 10:33:29.616984 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:29.616950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw" event={"ID":"ff411c05-36d3-41db-b8d7-a49afede66eb","Type":"ContainerStarted","Data":"ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0"}
Jul 29 10:33:36.652304 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:36.652268    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerID="366435f3e1b7ee1954317133b0fba47b9ed142d1c50a5e4e98bd5d821cb59e5d" exitCode=0
Jul 29 10:33:36.652718 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:36.652363    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw" event={"ID":"ff411c05-36d3-41db-b8d7-a49afede66eb","Type":"ContainerDied","Data":"366435f3e1b7ee1954317133b0fba47b9ed142d1c50a5e4e98bd5d821cb59e5d"}
Jul 29 10:33:39.666093 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:39.666041    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerID="04f27e7938f87b351f7f3ea42d20acfdfb710c776697560b34c29a4c493a9595" exitCode=0
Jul 29 10:33:39.666093 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:39.666094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw" event={"ID":"ff411c05-36d3-41db-b8d7-a49afede66eb","Type":"ContainerDied","Data":"04f27e7938f87b351f7f3ea42d20acfdfb710c776697560b34c29a4c493a9595"}
Jul 29 10:33:45.905344 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:45.905317    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:33:45.908097 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:45.905392    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:33:45.915839 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:45.915812    2574 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 10:33:46.703491 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:46.703456    2574 generic.go:356] "Generic (PLEG): container finished" podID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerID="7d0c6c0ec269c93d885ed888c29b4c4e703cffde43b5aef22be063a60bb5edaa" exitCode=0
Jul 29 10:33:46.703659 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:46.703492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw" event={"ID":"ff411c05-36d3-41db-b8d7-a49afede66eb","Type":"ContainerDied","Data":"7d0c6c0ec269c93d885ed888c29b4c4e703cffde43b5aef22be063a60bb5edaa"}
Jul 29 10:33:47.835208 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.835176    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:47.975074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.974986    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjl49\" (UniqueName: \"kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49\") pod \"ff411c05-36d3-41db-b8d7-a49afede66eb\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") "
Jul 29 10:33:47.975258 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.975085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle\") pod \"ff411c05-36d3-41db-b8d7-a49afede66eb\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") "
Jul 29 10:33:47.975258 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.975116    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util\") pod \"ff411c05-36d3-41db-b8d7-a49afede66eb\" (UID: \"ff411c05-36d3-41db-b8d7-a49afede66eb\") "
Jul 29 10:33:47.980678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.978447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49" (OuterVolumeSpecName: "kube-api-access-jjl49") pod "ff411c05-36d3-41db-b8d7-a49afede66eb" (UID: "ff411c05-36d3-41db-b8d7-a49afede66eb"). InnerVolumeSpecName "kube-api-access-jjl49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:33:47.980678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.978894    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle" (OuterVolumeSpecName: "bundle") pod "ff411c05-36d3-41db-b8d7-a49afede66eb" (UID: "ff411c05-36d3-41db-b8d7-a49afede66eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:33:47.983586 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:47.983555    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util" (OuterVolumeSpecName: "util") pod "ff411c05-36d3-41db-b8d7-a49afede66eb" (UID: "ff411c05-36d3-41db-b8d7-a49afede66eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:33:48.076761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.076723    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:33:48.076761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.076752    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff411c05-36d3-41db-b8d7-a49afede66eb-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:33:48.076761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.076762    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjl49\" (UniqueName: \"kubernetes.io/projected/ff411c05-36d3-41db-b8d7-a49afede66eb-kube-api-access-jjl49\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:33:48.713464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.713383    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw"
Jul 29 10:33:48.713464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.713391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5tbkhw" event={"ID":"ff411c05-36d3-41db-b8d7-a49afede66eb","Type":"ContainerDied","Data":"ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0"}
Jul 29 10:33:48.713464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:48.713436    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca87806c975d48e1584f1d7eadb11e10bbf6f5fabde597db1bbb01f3eaff4ae0"
Jul 29 10:33:51.133804 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.133763    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"]
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134026    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="pull"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134036    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="pull"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134055    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="extract"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134060    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="extract"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134068    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="util"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134073    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="util"
Jul 29 10:33:51.134182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.134113    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff411c05-36d3-41db-b8d7-a49afede66eb" containerName="extract"
Jul 29 10:33:51.176455 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.176416    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"]
Jul 29 10:33:51.176596 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.176543    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.178861 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.178831    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 29 10:33:51.179015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.178866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 29 10:33:51.179015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.178892    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-s5h9l\""
Jul 29 10:33:51.179015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.178898    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 29 10:33:51.306651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.306617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnvhr\" (UniqueName: \"kubernetes.io/projected/eb8d5a90-4065-43c6-84db-874b5cca926f-kube-api-access-dnvhr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.306651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.306656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb8d5a90-4065-43c6-84db-874b5cca926f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.408150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.408062    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnvhr\" (UniqueName: \"kubernetes.io/projected/eb8d5a90-4065-43c6-84db-874b5cca926f-kube-api-access-dnvhr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.408150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.408107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb8d5a90-4065-43c6-84db-874b5cca926f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.410625 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.410604    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb8d5a90-4065-43c6-84db-874b5cca926f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.417459 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.417434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnvhr\" (UniqueName: \"kubernetes.io/projected/eb8d5a90-4065-43c6-84db-874b5cca926f-kube-api-access-dnvhr\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b\" (UID: \"eb8d5a90-4065-43c6-84db-874b5cca926f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.486875 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.486835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:33:51.638869 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.638836    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"]
Jul 29 10:33:51.643273 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:33:51.643248    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8d5a90_4065_43c6_84db_874b5cca926f.slice/crio-4202f5a4355fba95012dffcf5c6c5480ca77aa0d5bf394fcd08f1e17d683d623 WatchSource:0}: Error finding container 4202f5a4355fba95012dffcf5c6c5480ca77aa0d5bf394fcd08f1e17d683d623: Status 404 returned error can't find the container with id 4202f5a4355fba95012dffcf5c6c5480ca77aa0d5bf394fcd08f1e17d683d623
Jul 29 10:33:51.645319 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.645299    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:33:51.726926 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:33:51.726844    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b" event={"ID":"eb8d5a90-4065-43c6-84db-874b5cca926f","Type":"ContainerStarted","Data":"4202f5a4355fba95012dffcf5c6c5480ca77aa0d5bf394fcd08f1e17d683d623"}
Jul 29 10:34:04.781084 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:04.781039    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b" event={"ID":"eb8d5a90-4065-43c6-84db-874b5cca926f","Type":"ContainerStarted","Data":"5e7b52ec74492cbae2b2c92a037ac6a378491779bba6957a34127965a7dbb165"}
Jul 29 10:34:04.781506 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:04.781122    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:34:04.805166 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:04.804695    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b" podStartSLOduration=0.853919065 podStartE2EDuration="13.804550721s" totalImagesPullingTime="12.950631656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:34:04.799935908 +0000 UTC m=+320.980317788" watchObservedRunningTime="2026-07-29 10:34:04.804550721 +0000 UTC m=+320.984932610"
Jul 29 10:34:05.360710 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.360676    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-vtqr8"]
Jul 29 10:34:05.362811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.362795    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.364961 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.364940    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-fxdrw\""
Jul 29 10:34:05.365337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.365321    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 29 10:34:05.365337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.365335    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 29 10:34:05.373800 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.373776    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-vtqr8"]
Jul 29 10:34:05.422580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.422536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhrj\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-kube-api-access-jnhrj\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.422757 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.422600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1e4e8ea9-d469-41f7-9f2b-de208e403305-cabundle0\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.422757 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.422713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.523996 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.523962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.524163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.524004    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhrj\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-kube-api-access-jnhrj\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.524163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.524050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1e4e8ea9-d469-41f7-9f2b-de208e403305-cabundle0\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.524163 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.524127    2574 secret.go:281] references non-existent secret key: ca.crt
Jul 29 10:34:05.524163 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.524149    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 29 10:34:05.524163 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.524162    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-vtqr8: references non-existent secret key: ca.crt
Jul 29 10:34:05.524391 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.524270    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates podName:1e4e8ea9-d469-41f7-9f2b-de208e403305 nodeName:}" failed. No retries permitted until 2026-07-29 10:34:06.024249311 +0000 UTC m=+322.204631181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates") pod "keda-operator-77f648d694-vtqr8" (UID: "1e4e8ea9-d469-41f7-9f2b-de208e403305") : references non-existent secret key: ca.crt
Jul 29 10:34:05.524696 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.524680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1e4e8ea9-d469-41f7-9f2b-de208e403305-cabundle0\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.532416 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.532394    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhrj\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-kube-api-access-jnhrj\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:05.556884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.556856    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"]
Jul 29 10:34:05.559125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.559109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.561196 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.561173    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 29 10:34:05.569491 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.569462    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"]
Jul 29 10:34:05.625319 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.625203    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2339e349-82c9-4b1a-bf0e-88584f1b27bc-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.625319 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.625311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.625507 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.625327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh929\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-kube-api-access-wh929\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.725779 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.725747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2339e349-82c9-4b1a-bf0e-88584f1b27bc-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.725978 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.725820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.725978 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.725841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh929\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-kube-api-access-wh929\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.725978 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.725969    2574 secret.go:281] references non-existent secret key: tls.crt
Jul 29 10:34:05.726145 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.725989    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 29 10:34:05.726145 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.726011    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw: references non-existent secret key: tls.crt
Jul 29 10:34:05.726145 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:05.726080    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates podName:2339e349-82c9-4b1a-bf0e-88584f1b27bc nodeName:}" failed. No retries permitted until 2026-07-29 10:34:06.226061201 +0000 UTC m=+322.406443076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates") pod "keda-metrics-apiserver-55557df8c-hkqpw" (UID: "2339e349-82c9-4b1a-bf0e-88584f1b27bc") : references non-existent secret key: tls.crt
Jul 29 10:34:05.726348 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.726301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/2339e349-82c9-4b1a-bf0e-88584f1b27bc-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:05.736658 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:05.736638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh929\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-kube-api-access-wh929\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:06.027621 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.027586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:06.028019 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.027719    2574 secret.go:281] references non-existent secret key: ca.crt
Jul 29 10:34:06.028019 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.027737    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 29 10:34:06.028019 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.027745    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-vtqr8: references non-existent secret key: ca.crt
Jul 29 10:34:06.028019 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.027798    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates podName:1e4e8ea9-d469-41f7-9f2b-de208e403305 nodeName:}" failed. No retries permitted until 2026-07-29 10:34:07.02778385 +0000 UTC m=+323.208165708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates") pod "keda-operator-77f648d694-vtqr8" (UID: "1e4e8ea9-d469-41f7-9f2b-de208e403305") : references non-existent secret key: ca.crt
Jul 29 10:34:06.229006 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.228965    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:06.229183 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.229117    2574 secret.go:281] references non-existent secret key: tls.crt
Jul 29 10:34:06.229183 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.229133    2574 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 29 10:34:06.229183 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.229152    2574 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw: references non-existent secret key: tls.crt
Jul 29 10:34:06.229325 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:34:06.229207    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates podName:2339e349-82c9-4b1a-bf0e-88584f1b27bc nodeName:}" failed. No retries permitted until 2026-07-29 10:34:07.229190114 +0000 UTC m=+323.409571993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates") pod "keda-metrics-apiserver-55557df8c-hkqpw" (UID: "2339e349-82c9-4b1a-bf0e-88584f1b27bc") : references non-existent secret key: tls.crt
Jul 29 10:34:06.322955 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.322874    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t5v4l"]
Jul 29 10:34:06.324938 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.324922    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.327035 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.327017    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 29 10:34:06.329326 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.329302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-certificates\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.329420 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.329370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqk8\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-kube-api-access-zvqk8\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.342908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.342884    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t5v4l"]
Jul 29 10:34:06.430174 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.430142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqk8\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-kube-api-access-zvqk8\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.430384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.430210    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-certificates\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.432894 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.432866    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-certificates\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.438093 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.438058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqk8\" (UniqueName: \"kubernetes.io/projected/7f262588-1a87-4e99-8d77-a2bcb9bfc31c-kube-api-access-zvqk8\") pod \"keda-admission-56f9798bb4-t5v4l\" (UID: \"7f262588-1a87-4e99-8d77-a2bcb9bfc31c\") " pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.635008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.634682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:06.787989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:06.787912    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-t5v4l"]
Jul 29 10:34:06.792059 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:34:06.792020    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f262588_1a87_4e99_8d77_a2bcb9bfc31c.slice/crio-5f81917e2bc5d9a2b13d86248a656ae6d4af8f1ab1dbfc8db1db5b81924d2e8c WatchSource:0}: Error finding container 5f81917e2bc5d9a2b13d86248a656ae6d4af8f1ab1dbfc8db1db5b81924d2e8c: Status 404 returned error can't find the container with id 5f81917e2bc5d9a2b13d86248a656ae6d4af8f1ab1dbfc8db1db5b81924d2e8c
Jul 29 10:34:07.036007 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.035971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:07.038548 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.038519    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1e4e8ea9-d469-41f7-9f2b-de208e403305-certificates\") pod \"keda-operator-77f648d694-vtqr8\" (UID: \"1e4e8ea9-d469-41f7-9f2b-de208e403305\") " pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:07.173267 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.173211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:07.237632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.237559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:07.240169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.240134    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2339e349-82c9-4b1a-bf0e-88584f1b27bc-certificates\") pod \"keda-metrics-apiserver-55557df8c-hkqpw\" (UID: \"2339e349-82c9-4b1a-bf0e-88584f1b27bc\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:07.316130 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.316050    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-vtqr8"]
Jul 29 10:34:07.319596 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:34:07.319566    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4e8ea9_d469_41f7_9f2b_de208e403305.slice/crio-0eff2013cb188b2d4a4cad9b152be7a7b673888761d7797ac2393e33d2511c6d WatchSource:0}: Error finding container 0eff2013cb188b2d4a4cad9b152be7a7b673888761d7797ac2393e33d2511c6d: Status 404 returned error can't find the container with id 0eff2013cb188b2d4a4cad9b152be7a7b673888761d7797ac2393e33d2511c6d
Jul 29 10:34:07.371758 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.371733    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:07.508455 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.508425    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"]
Jul 29 10:34:07.511538 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:34:07.511513    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2339e349_82c9_4b1a_bf0e_88584f1b27bc.slice/crio-474ba919a73a60acb0cc7017ee029a7bc40855f55bcc40fc14acf78efa68545d WatchSource:0}: Error finding container 474ba919a73a60acb0cc7017ee029a7bc40855f55bcc40fc14acf78efa68545d: Status 404 returned error can't find the container with id 474ba919a73a60acb0cc7017ee029a7bc40855f55bcc40fc14acf78efa68545d
Jul 29 10:34:07.796435 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.796396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l" event={"ID":"7f262588-1a87-4e99-8d77-a2bcb9bfc31c","Type":"ContainerStarted","Data":"5f81917e2bc5d9a2b13d86248a656ae6d4af8f1ab1dbfc8db1db5b81924d2e8c"}
Jul 29 10:34:07.797520 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.797488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-vtqr8" event={"ID":"1e4e8ea9-d469-41f7-9f2b-de208e403305","Type":"ContainerStarted","Data":"0eff2013cb188b2d4a4cad9b152be7a7b673888761d7797ac2393e33d2511c6d"}
Jul 29 10:34:07.798694 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:07.798675    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw" event={"ID":"2339e349-82c9-4b1a-bf0e-88584f1b27bc","Type":"ContainerStarted","Data":"474ba919a73a60acb0cc7017ee029a7bc40855f55bcc40fc14acf78efa68545d"}
Jul 29 10:34:10.813567 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:10.813520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-vtqr8" event={"ID":"1e4e8ea9-d469-41f7-9f2b-de208e403305","Type":"ContainerStarted","Data":"9650f332ebbb70260aa09a5a4ee765e35b983f6c56f4222eb84a10e6cdae436f"}
Jul 29 10:34:10.813938 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:10.813642    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:10.829791 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:10.829740    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-vtqr8" podStartSLOduration=2.937596187 podStartE2EDuration="5.829721472s" totalImagesPullingTime="2.892125285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:34:10.828892657 +0000 UTC m=+327.009274537" watchObservedRunningTime="2026-07-29 10:34:10.829721472 +0000 UTC m=+327.010103354"
Jul 29 10:34:14.832484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.832450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw" event={"ID":"2339e349-82c9-4b1a-bf0e-88584f1b27bc","Type":"ContainerStarted","Data":"7d7509ef8a6d5559aadc96e90e5d3dabedecd778306804fa57ab089de3747241"}
Jul 29 10:34:14.832929 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.832573    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:14.833993 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.833970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l" event={"ID":"7f262588-1a87-4e99-8d77-a2bcb9bfc31c","Type":"ContainerStarted","Data":"bda9985af810d210acb1e50f7a93bc596d12f7fb9723108520e5be693b0aa371"}
Jul 29 10:34:14.834125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.834111    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:34:14.847161 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.847103    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw" podStartSLOduration=3.444806924 podStartE2EDuration="9.847085015s" totalImagesPullingTime="6.402278091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:34:14.846609461 +0000 UTC m=+331.026991342" watchObservedRunningTime="2026-07-29 10:34:14.847085015 +0000 UTC m=+331.027466897"
Jul 29 10:34:14.862232 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:14.862179    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l" podStartSLOduration=0.965873993 podStartE2EDuration="8.862163435s" totalImagesPullingTime="7.896289442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:34:14.860710985 +0000 UTC m=+331.041092866" watchObservedRunningTime="2026-07-29 10:34:14.862163435 +0000 UTC m=+331.042545314"
Jul 29 10:34:25.788132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:25.788098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-zlc6b"
Jul 29 10:34:25.844115 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:25.844084    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-hkqpw"
Jul 29 10:34:31.821441 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:31.821410    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-vtqr8"
Jul 29 10:34:35.841441 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:34:35.841353    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-t5v4l"
Jul 29 10:35:23.466253 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.466198    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 10:35:23.468574 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.468552    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:23.470763 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.470738    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-96ksr\""
Jul 29 10:35:23.470905 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.470775    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 29 10:35:23.471159 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.471136    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 29 10:35:23.471298 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.471141    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 29 10:35:23.478733 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.478705    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 10:35:23.558998 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.558963    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jnw\" (UniqueName: \"kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:23.559160 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.559024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:23.660196 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.660151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:23.660398 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.660263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jnw\" (UniqueName: \"kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:23.660398 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:35:23.660329    2574 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 29 10:35:23.660478 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:35:23.660412    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert podName:e0741931-2ec6-4c8f-bf3b-de4f6441f8c4 nodeName:}" failed. No retries permitted until 2026-07-29 10:35:24.160395577 +0000 UTC m=+400.340777439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert") pod "llmisvc-controller-manager-67c47db88-fw5qc" (UID: "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4") : secret "llmisvc-webhook-server-cert" not found
Jul 29 10:35:23.669392 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:23.669354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jnw\" (UniqueName: \"kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:24.164154 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:24.164111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:24.164361 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:35:24.164281    2574 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 29 10:35:24.164361 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:35:24.164339    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert podName:e0741931-2ec6-4c8f-bf3b-de4f6441f8c4 nodeName:}" failed. No retries permitted until 2026-07-29 10:35:25.164325323 +0000 UTC m=+401.344707181 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert") pod "llmisvc-controller-manager-67c47db88-fw5qc" (UID: "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4") : secret "llmisvc-webhook-server-cert" not found
Jul 29 10:35:25.173079 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:25.173042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:25.175587 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:25.175565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"llmisvc-controller-manager-67c47db88-fw5qc\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") " pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:25.279452 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:25.279422    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:25.416305 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:25.416273    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 10:35:25.420074 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:35:25.420040    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0741931_2ec6_4c8f_bf3b_de4f6441f8c4.slice/crio-9c893595dc5ab0eae916ecc8f2600af17a7772054ddda7415dd1e619cf902273 WatchSource:0}: Error finding container 9c893595dc5ab0eae916ecc8f2600af17a7772054ddda7415dd1e619cf902273: Status 404 returned error can't find the container with id 9c893595dc5ab0eae916ecc8f2600af17a7772054ddda7415dd1e619cf902273
Jul 29 10:35:26.135633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:26.135583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" event={"ID":"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4","Type":"ContainerStarted","Data":"9c893595dc5ab0eae916ecc8f2600af17a7772054ddda7415dd1e619cf902273"}
Jul 29 10:35:29.150551 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:29.150505    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" event={"ID":"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4","Type":"ContainerStarted","Data":"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"}
Jul 29 10:35:29.151100 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:29.150667    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:35:29.167706 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:35:29.167641    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" podStartSLOduration=3.240185536 podStartE2EDuration="6.167620125s" totalImagesPullingTime="2.927434589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:35:29.165804735 +0000 UTC m=+405.346186616" watchObservedRunningTime="2026-07-29 10:35:29.167620125 +0000 UTC m=+405.348002005"
Jul 29 10:36:00.158013 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:00.157981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 10:36:07.550054 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.549973    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"]
Jul 29 10:36:07.553485 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.553469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.555651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.555628    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 10:36:07.555651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.555630    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhmdt\""
Jul 29 10:36:07.556130 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.556113    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 10:36:07.560811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.560477    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"]
Jul 29 10:36:07.731611 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.731570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.731611 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.731611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjd6\" (UniqueName: \"kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.731825 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.731716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.832643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.832556    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.832643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.832612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.832643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.832632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjd6\" (UniqueName: \"kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.832947 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.832924    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.833029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.833008    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.841666 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.841641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjd6\" (UniqueName: \"kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.863513 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.863483    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:07.999483 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:07.999449    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"]
Jul 29 10:36:08.002277 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:36:08.002251    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb8a034_c56f_44bb_8654_d30983eacbc9.slice/crio-97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f WatchSource:0}: Error finding container 97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f: Status 404 returned error can't find the container with id 97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f
Jul 29 10:36:08.315955 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:08.315924    2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerID="f320fbcc67cdca8cb413b91dbc281ff4f15a4c411e29c19d06fcbaa93b0319a3" exitCode=0
Jul 29 10:36:08.316170 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:08.315967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp" event={"ID":"7bb8a034-c56f-44bb-8654-d30983eacbc9","Type":"ContainerDied","Data":"f320fbcc67cdca8cb413b91dbc281ff4f15a4c411e29c19d06fcbaa93b0319a3"}
Jul 29 10:36:08.316170 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:08.316005    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp" event={"ID":"7bb8a034-c56f-44bb-8654-d30983eacbc9","Type":"ContainerStarted","Data":"97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f"}
Jul 29 10:36:11.331336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:11.331304    2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerID="c54dd6468e8217372365ada2c935297102bdce87fe9df3e475dc7d4376f8ed18" exitCode=0
Jul 29 10:36:11.331734 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:11.331373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp" event={"ID":"7bb8a034-c56f-44bb-8654-d30983eacbc9","Type":"ContainerDied","Data":"c54dd6468e8217372365ada2c935297102bdce87fe9df3e475dc7d4376f8ed18"}
Jul 29 10:36:12.337932 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:12.337898    2574 generic.go:356] "Generic (PLEG): container finished" podID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerID="0eda8aa2bc98c9eb00857c4001bcf27f57bbee0da4dd9db4d84d18805889e0d7" exitCode=0
Jul 29 10:36:12.341013 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:12.340982    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp" event={"ID":"7bb8a034-c56f-44bb-8654-d30983eacbc9","Type":"ContainerDied","Data":"0eda8aa2bc98c9eb00857c4001bcf27f57bbee0da4dd9db4d84d18805889e0d7"}
Jul 29 10:36:13.472829 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.472798    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:13.476125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.476103    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle\") pod \"7bb8a034-c56f-44bb-8654-d30983eacbc9\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") "
Jul 29 10:36:13.476287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.476144    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfjd6\" (UniqueName: \"kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6\") pod \"7bb8a034-c56f-44bb-8654-d30983eacbc9\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") "
Jul 29 10:36:13.476287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.476195    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util\") pod \"7bb8a034-c56f-44bb-8654-d30983eacbc9\" (UID: \"7bb8a034-c56f-44bb-8654-d30983eacbc9\") "
Jul 29 10:36:13.476961 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.476931    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle" (OuterVolumeSpecName: "bundle") pod "7bb8a034-c56f-44bb-8654-d30983eacbc9" (UID: "7bb8a034-c56f-44bb-8654-d30983eacbc9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:36:13.478335 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.478312    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6" (OuterVolumeSpecName: "kube-api-access-wfjd6") pod "7bb8a034-c56f-44bb-8654-d30983eacbc9" (UID: "7bb8a034-c56f-44bb-8654-d30983eacbc9"). InnerVolumeSpecName "kube-api-access-wfjd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:36:13.483390 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.483366    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util" (OuterVolumeSpecName: "util") pod "7bb8a034-c56f-44bb-8654-d30983eacbc9" (UID: "7bb8a034-c56f-44bb-8654-d30983eacbc9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:36:13.577008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.576973    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:13.577008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.577003    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjd6\" (UniqueName: \"kubernetes.io/projected/7bb8a034-c56f-44bb-8654-d30983eacbc9-kube-api-access-wfjd6\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:13.577008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:13.577013    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7bb8a034-c56f-44bb-8654-d30983eacbc9-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:14.348860 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:14.348830    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp"
Jul 29 10:36:14.349070 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:14.348830    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402j8jp" event={"ID":"7bb8a034-c56f-44bb-8654-d30983eacbc9","Type":"ContainerDied","Data":"97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f"}
Jul 29 10:36:14.349070 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:14.348956    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97575292316c0a8f43e1d052f6bc321d36a9c3d31c4dfe763eea8779ee2e678f"
Jul 29 10:36:29.675874 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.675838    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"]
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676148    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="extract"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676159    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="extract"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676171    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="util"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676177    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="util"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676193    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="pull"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676198    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="pull"
Jul 29 10:36:29.676264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.676264    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bb8a034-c56f-44bb-8654-d30983eacbc9" containerName="extract"
Jul 29 10:36:29.680031 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.680014    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.682479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.682456    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 10:36:29.682602 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.682492    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 10:36:29.682940 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.682927    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhmdt\""
Jul 29 10:36:29.687811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.687789    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"]
Jul 29 10:36:29.694155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.694129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.694330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.694180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hldg9\" (UniqueName: \"kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.694330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.694200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.794687 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.794643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hldg9\" (UniqueName: \"kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.794687 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.794688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.794896 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.794753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.795072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.795054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.795156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.795137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.804716 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.804678    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hldg9\" (UniqueName: \"kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:29.990950 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:29.990855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:30.147081 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:30.147045    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"]
Jul 29 10:36:30.150852 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:36:30.150818    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa01d347_2571_4f11_bda6_65dce35d6daa.slice/crio-00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10 WatchSource:0}: Error finding container 00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10: Status 404 returned error can't find the container with id 00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10
Jul 29 10:36:30.420969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:30.420930    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerID="419115b803868e204c2be6bc7201e2853734157b6ce3bd05ef73ade1df7f84d4" exitCode=0
Jul 29 10:36:30.421156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:30.421010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg" event={"ID":"aa01d347-2571-4f11-bda6-65dce35d6daa","Type":"ContainerDied","Data":"419115b803868e204c2be6bc7201e2853734157b6ce3bd05ef73ade1df7f84d4"}
Jul 29 10:36:30.421156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:30.421063    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg" event={"ID":"aa01d347-2571-4f11-bda6-65dce35d6daa","Type":"ContainerStarted","Data":"00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10"}
Jul 29 10:36:34.443758 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:34.443719    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerID="3e43b2cbdad5db8dfef3ddfadcf940657f1576bc5a8c1b7fe297ed114143cc9b" exitCode=0
Jul 29 10:36:34.444148 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:34.443779    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg" event={"ID":"aa01d347-2571-4f11-bda6-65dce35d6daa","Type":"ContainerDied","Data":"3e43b2cbdad5db8dfef3ddfadcf940657f1576bc5a8c1b7fe297ed114143cc9b"}
Jul 29 10:36:35.451003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:35.450967    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerID="5b776537d1bae21488e906e0b96fe0acea018ab4809be69eb10652ef6be77f12" exitCode=0
Jul 29 10:36:35.451431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:35.451041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg" event={"ID":"aa01d347-2571-4f11-bda6-65dce35d6daa","Type":"ContainerDied","Data":"5b776537d1bae21488e906e0b96fe0acea018ab4809be69eb10652ef6be77f12"}
Jul 29 10:36:36.585315 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.585283    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:36.651441 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.651411    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util\") pod \"aa01d347-2571-4f11-bda6-65dce35d6daa\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") "
Jul 29 10:36:36.651605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.651481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle\") pod \"aa01d347-2571-4f11-bda6-65dce35d6daa\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") "
Jul 29 10:36:36.651605 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.651533    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hldg9\" (UniqueName: \"kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9\") pod \"aa01d347-2571-4f11-bda6-65dce35d6daa\" (UID: \"aa01d347-2571-4f11-bda6-65dce35d6daa\") "
Jul 29 10:36:36.651893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.651871    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle" (OuterVolumeSpecName: "bundle") pod "aa01d347-2571-4f11-bda6-65dce35d6daa" (UID: "aa01d347-2571-4f11-bda6-65dce35d6daa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:36:36.653747 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.653720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9" (OuterVolumeSpecName: "kube-api-access-hldg9") pod "aa01d347-2571-4f11-bda6-65dce35d6daa" (UID: "aa01d347-2571-4f11-bda6-65dce35d6daa"). InnerVolumeSpecName "kube-api-access-hldg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:36:36.655789 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.655768    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util" (OuterVolumeSpecName: "util") pod "aa01d347-2571-4f11-bda6-65dce35d6daa" (UID: "aa01d347-2571-4f11-bda6-65dce35d6daa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:36:36.752383 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.752293    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hldg9\" (UniqueName: \"kubernetes.io/projected/aa01d347-2571-4f11-bda6-65dce35d6daa-kube-api-access-hldg9\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:36.752383 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.752320    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:36.752383 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:36.752330    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa01d347-2571-4f11-bda6-65dce35d6daa-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:36:37.461593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:37.461565    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg"
Jul 29 10:36:37.461593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:37.461585    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcmrzg" event={"ID":"aa01d347-2571-4f11-bda6-65dce35d6daa","Type":"ContainerDied","Data":"00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10"}
Jul 29 10:36:37.461846 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:37.461629    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ab148ee35bdf79249d733ed49246e8e93d68cac5f7cb75493c5fdbaf7fbb10"
Jul 29 10:36:56.633768 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.633731    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"]
Jul 29 10:36:56.637118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637086    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="pull"
Jul 29 10:36:56.637326 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637313    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="pull"
Jul 29 10:36:56.637417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637407    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="extract"
Jul 29 10:36:56.637484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637475    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="extract"
Jul 29 10:36:56.637595 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637585    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="util"
Jul 29 10:36:56.637668 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637660    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="util"
Jul 29 10:36:56.637910 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.637897    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa01d347-2571-4f11-bda6-65dce35d6daa" containerName="extract"
Jul 29 10:36:56.647631 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.647600    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"]
Jul 29 10:36:56.647784 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.647713    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.650110 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.650082    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 10:36:56.650385 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.650370    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 10:36:56.650875 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.650850    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 10:36:56.650987 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.650896    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6tm4x\""
Jul 29 10:36:56.650987 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.650896    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 10:36:56.651114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.651074    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 10:36:56.722656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.722616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.722656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.722660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27cbc149-314b-4b4d-acd6-52452642acd5-manager-config\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.722934 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.722687    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.722934 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.722769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chs6v\" (UniqueName: \"kubernetes.io/projected/27cbc149-314b-4b4d-acd6-52452642acd5-kube-api-access-chs6v\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.823734 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.823694    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.823937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.823760    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chs6v\" (UniqueName: \"kubernetes.io/projected/27cbc149-314b-4b4d-acd6-52452642acd5-kube-api-access-chs6v\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.823937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.823843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.823937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.823869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27cbc149-314b-4b4d-acd6-52452642acd5-manager-config\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.824518 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.824494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/27cbc149-314b-4b4d-acd6-52452642acd5-manager-config\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.826380 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.826360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-metrics-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.826455 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.826394    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27cbc149-314b-4b4d-acd6-52452642acd5-cert\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.835458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.835433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chs6v\" (UniqueName: \"kubernetes.io/projected/27cbc149-314b-4b4d-acd6-52452642acd5-kube-api-access-chs6v\") pod \"lws-controller-manager-5c847d9c7c-9t4vb\" (UID: \"27cbc149-314b-4b4d-acd6-52452642acd5\") " pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:56.958844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:56.958750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:36:57.115603 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:57.115571    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"]
Jul 29 10:36:57.118957 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:36:57.118931    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cbc149_314b_4b4d_acd6_52452642acd5.slice/crio-81bf24573f7c4c6f00a52c1b66773e44fcd5b199d7fe993f5de07adbaf975d84 WatchSource:0}: Error finding container 81bf24573f7c4c6f00a52c1b66773e44fcd5b199d7fe993f5de07adbaf975d84: Status 404 returned error can't find the container with id 81bf24573f7c4c6f00a52c1b66773e44fcd5b199d7fe993f5de07adbaf975d84
Jul 29 10:36:57.553714 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:36:57.553677    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb" event={"ID":"27cbc149-314b-4b4d-acd6-52452642acd5","Type":"ContainerStarted","Data":"81bf24573f7c4c6f00a52c1b66773e44fcd5b199d7fe993f5de07adbaf975d84"}
Jul 29 10:37:00.568317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:00.568282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb" event={"ID":"27cbc149-314b-4b4d-acd6-52452642acd5","Type":"ContainerStarted","Data":"3020b74a3d5988f5965b31daa110658e194718b540bd799bb71da433d13ddd72"}
Jul 29 10:37:00.568782 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:00.568407    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:37:00.585989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:00.585938    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb" podStartSLOduration=1.746316311 podStartE2EDuration="4.58592289s" totalImagesPullingTime="2.839606579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:00.584285839 +0000 UTC m=+496.764667719" watchObservedRunningTime="2026-07-29 10:37:00.58592289 +0000 UTC m=+496.766304769"
Jul 29 10:37:11.576584 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:11.576550    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c847d9c7c-9t4vb"
Jul 29 10:37:14.156309 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.156268    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"]
Jul 29 10:37:14.159648 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.159630    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.162003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.161982    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 10:37:14.162144 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.161985    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 10:37:14.162371 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.162355    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mhmdt\""
Jul 29 10:37:14.170844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.170815    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"]
Jul 29 10:37:14.180511 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.180484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.180668 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.180541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.180668 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.180598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22djv\" (UniqueName: \"kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.256476 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.256435    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"]
Jul 29 10:37:14.259849 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.259831    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.267923 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.267898    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"]
Jul 29 10:37:14.281862 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.281831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.282066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.281875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.282066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.281900    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22djv\" (UniqueName: \"kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.282066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.281964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.282066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.282014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.282066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.282044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bftxl\" (UniqueName: \"kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.282367 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.282329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.282415 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.282339    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.290176 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.290151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22djv\" (UniqueName: \"kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.352333 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.352293    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"]
Jul 29 10:37:14.355804 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.355782    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.364325 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.364296    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"]
Jul 29 10:37:14.382600 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.382600 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.382845 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.382845 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.382845 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382797    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bftxl\" (UniqueName: \"kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.382997 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vm65\" (UniqueName: \"kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.382997 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.382984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.383081 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.383026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.390608 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.390583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bftxl\" (UniqueName: \"kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.456430 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.456349    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"]
Jul 29 10:37:14.459893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.459875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.467504 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.467307    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"]
Jul 29 10:37:14.469368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.469342    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:14.483857 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.483827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm65\" (UniqueName: \"kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.484012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.483881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmpfc\" (UniqueName: \"kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.484012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.483918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.484012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.483950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.484190 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.484071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.484190 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.484121    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.484580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.484557    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.484680 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.484639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.491677 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.491652    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm65\" (UniqueName: \"kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.569704 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.569671    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:14.585593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.585525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmpfc\" (UniqueName: \"kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.585776 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.585661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.585776 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.585700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.586097 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.586075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.586182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.586132    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.595082 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.595048    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmpfc\" (UniqueName: \"kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.615308 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.615194    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"]
Jul 29 10:37:14.618726 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:37:14.618693    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714748e4_fdc8_4325_8063_cded5b69c6c8.slice/crio-c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e WatchSource:0}: Error finding container c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e: Status 404 returned error can't find the container with id c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e
Jul 29 10:37:14.644706 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.644665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz" event={"ID":"714748e4-fdc8-4325-8063-cded5b69c6c8","Type":"ContainerStarted","Data":"c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e"}
Jul 29 10:37:14.666646 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.666618    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:14.724572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.724437    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"]
Jul 29 10:37:14.729363 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:37:14.729329    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96b743c_df3c_46a1_8a3a_7fcdfe5c2712.slice/crio-d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21 WatchSource:0}: Error finding container d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21: Status 404 returned error can't find the container with id d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21
Jul 29 10:37:14.770638 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.770601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:14.825676 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.825647    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"]
Jul 29 10:37:14.830149 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:37:14.830119    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47c56db_97b1_429c_bee2_2d24cf253186.slice/crio-d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1 WatchSource:0}: Error finding container d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1: Status 404 returned error can't find the container with id d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1
Jul 29 10:37:14.931000 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:14.930954    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"]
Jul 29 10:37:14.942303 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:37:14.942269    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb59c1858_2841_4c89_9925_e3a65595aebb.slice/crio-2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7 WatchSource:0}: Error finding container 2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7: Status 404 returned error can't find the container with id 2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7
Jul 29 10:37:15.650490 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.650460    2574 generic.go:356] "Generic (PLEG): container finished" podID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerID="b210233825ef64d3af98e51d69c7ee97d3eed7243fed4993e4b6ef36cb1adf55" exitCode=0
Jul 29 10:37:15.650918 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.650535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz" event={"ID":"714748e4-fdc8-4325-8063-cded5b69c6c8","Type":"ContainerDied","Data":"b210233825ef64d3af98e51d69c7ee97d3eed7243fed4993e4b6ef36cb1adf55"}
Jul 29 10:37:15.652274 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.652248    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47c56db-97b1-429c-bee2-2d24cf253186" containerID="76006e5d26a9a096473803448cc6ae561cb069ede38fc0bc3ce4180564955e6e" exitCode=0
Jul 29 10:37:15.652374 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.652321    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4" event={"ID":"c47c56db-97b1-429c-bee2-2d24cf253186","Type":"ContainerDied","Data":"76006e5d26a9a096473803448cc6ae561cb069ede38fc0bc3ce4180564955e6e"}
Jul 29 10:37:15.652374 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.652371    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4" event={"ID":"c47c56db-97b1-429c-bee2-2d24cf253186","Type":"ContainerStarted","Data":"d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1"}
Jul 29 10:37:15.653833 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.653811    2574 generic.go:356] "Generic (PLEG): container finished" podID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerID="5ea718d6208c12e0d46f7f67008d8d66145c17ce927b1034a3646e117ded8706" exitCode=0
Jul 29 10:37:15.653910 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.653892    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws" event={"ID":"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712","Type":"ContainerDied","Data":"5ea718d6208c12e0d46f7f67008d8d66145c17ce927b1034a3646e117ded8706"}
Jul 29 10:37:15.653958 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.653935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws" event={"ID":"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712","Type":"ContainerStarted","Data":"d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21"}
Jul 29 10:37:15.655427 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.655410    2574 generic.go:356] "Generic (PLEG): container finished" podID="b59c1858-2841-4c89-9925-e3a65595aebb" containerID="3533995861ab170463aa5bad6d1cfe9d5e16a48a373590b2e077c5519b0350fb" exitCode=0
Jul 29 10:37:15.655484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.655454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j" event={"ID":"b59c1858-2841-4c89-9925-e3a65595aebb","Type":"ContainerDied","Data":"3533995861ab170463aa5bad6d1cfe9d5e16a48a373590b2e077c5519b0350fb"}
Jul 29 10:37:15.655484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:15.655481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j" event={"ID":"b59c1858-2841-4c89-9925-e3a65595aebb","Type":"ContainerStarted","Data":"2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7"}
Jul 29 10:37:16.661989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:16.661949    2574 generic.go:356] "Generic (PLEG): container finished" podID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerID="a19aeda0af112f5d742caa6788715b5a0435d917a4a022235451edccaef08c2f" exitCode=0
Jul 29 10:37:16.662438 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:16.662021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz" event={"ID":"714748e4-fdc8-4325-8063-cded5b69c6c8","Type":"ContainerDied","Data":"a19aeda0af112f5d742caa6788715b5a0435d917a4a022235451edccaef08c2f"}
Jul 29 10:37:16.663875 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:16.663854    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47c56db-97b1-429c-bee2-2d24cf253186" containerID="9c03dcfd3aac36f9be36f522cc2aa1ad41c41d23f229324b26f09f02ca0cd48b" exitCode=0
Jul 29 10:37:16.663972 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:16.663912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4" event={"ID":"c47c56db-97b1-429c-bee2-2d24cf253186","Type":"ContainerDied","Data":"9c03dcfd3aac36f9be36f522cc2aa1ad41c41d23f229324b26f09f02ca0cd48b"}
Jul 29 10:37:17.672320 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:17.672235    2574 generic.go:356] "Generic (PLEG): container finished" podID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerID="1e56fc575e4db461c0cfb3285f6a1d833d9caed6f175eae56b1b74f33e2d611c" exitCode=0
Jul 29 10:37:17.672320 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:17.672261    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz" event={"ID":"714748e4-fdc8-4325-8063-cded5b69c6c8","Type":"ContainerDied","Data":"1e56fc575e4db461c0cfb3285f6a1d833d9caed6f175eae56b1b74f33e2d611c"}
Jul 29 10:37:17.674570 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:17.674544    2574 generic.go:356] "Generic (PLEG): container finished" podID="c47c56db-97b1-429c-bee2-2d24cf253186" containerID="0731c9b7cbee3fa1931efe3915e3e5a149976211d22d5c3f03349153082231a6" exitCode=0
Jul 29 10:37:17.674730 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:17.674629    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4" event={"ID":"c47c56db-97b1-429c-bee2-2d24cf253186","Type":"ContainerDied","Data":"0731c9b7cbee3fa1931efe3915e3e5a149976211d22d5c3f03349153082231a6"}
Jul 29 10:37:18.815780 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.815751    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:18.824279 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.824250    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle\") pod \"714748e4-fdc8-4325-8063-cded5b69c6c8\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") "
Jul 29 10:37:18.824279 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.824283    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22djv\" (UniqueName: \"kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv\") pod \"714748e4-fdc8-4325-8063-cded5b69c6c8\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") "
Jul 29 10:37:18.824499 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.824323    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util\") pod \"714748e4-fdc8-4325-8063-cded5b69c6c8\" (UID: \"714748e4-fdc8-4325-8063-cded5b69c6c8\") "
Jul 29 10:37:18.824823 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.824798    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle" (OuterVolumeSpecName: "bundle") pod "714748e4-fdc8-4325-8063-cded5b69c6c8" (UID: "714748e4-fdc8-4325-8063-cded5b69c6c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:18.826919 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.826893    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv" (OuterVolumeSpecName: "kube-api-access-22djv") pod "714748e4-fdc8-4325-8063-cded5b69c6c8" (UID: "714748e4-fdc8-4325-8063-cded5b69c6c8"). InnerVolumeSpecName "kube-api-access-22djv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:37:18.830300 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.830270    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util" (OuterVolumeSpecName: "util") pod "714748e4-fdc8-4325-8063-cded5b69c6c8" (UID: "714748e4-fdc8-4325-8063-cded5b69c6c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:18.851860 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.851826    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:18.925437 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925402    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle\") pod \"c47c56db-97b1-429c-bee2-2d24cf253186\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") "
Jul 29 10:37:18.925614 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925470    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vm65\" (UniqueName: \"kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65\") pod \"c47c56db-97b1-429c-bee2-2d24cf253186\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") "
Jul 29 10:37:18.925614 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925500    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util\") pod \"c47c56db-97b1-429c-bee2-2d24cf253186\" (UID: \"c47c56db-97b1-429c-bee2-2d24cf253186\") "
Jul 29 10:37:18.925735 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925717    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:18.925819 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925741    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22djv\" (UniqueName: \"kubernetes.io/projected/714748e4-fdc8-4325-8063-cded5b69c6c8-kube-api-access-22djv\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:18.925819 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925756    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/714748e4-fdc8-4325-8063-cded5b69c6c8-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:18.925962 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.925938    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle" (OuterVolumeSpecName: "bundle") pod "c47c56db-97b1-429c-bee2-2d24cf253186" (UID: "c47c56db-97b1-429c-bee2-2d24cf253186"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:18.927884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.927859    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65" (OuterVolumeSpecName: "kube-api-access-6vm65") pod "c47c56db-97b1-429c-bee2-2d24cf253186" (UID: "c47c56db-97b1-429c-bee2-2d24cf253186"). InnerVolumeSpecName "kube-api-access-6vm65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:37:18.931460 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:18.931385    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util" (OuterVolumeSpecName: "util") pod "c47c56db-97b1-429c-bee2-2d24cf253186" (UID: "c47c56db-97b1-429c-bee2-2d24cf253186"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:19.027008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.026967    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vm65\" (UniqueName: \"kubernetes.io/projected/c47c56db-97b1-429c-bee2-2d24cf253186-kube-api-access-6vm65\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:19.027008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.027000    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:19.027008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.027010    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c47c56db-97b1-429c-bee2-2d24cf253186-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:19.687175 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.687142    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz"
Jul 29 10:37:19.687175 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.687159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c7ccxz" event={"ID":"714748e4-fdc8-4325-8063-cded5b69c6c8","Type":"ContainerDied","Data":"c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e"}
Jul 29 10:37:19.687417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.687209    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09564a77932471ffb2d880bab206a142f418fe6ec9605480faf6177b417f66e"
Jul 29 10:37:19.689031 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.689010    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4"
Jul 29 10:37:19.689136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.689013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbsr9r4" event={"ID":"c47c56db-97b1-429c-bee2-2d24cf253186","Type":"ContainerDied","Data":"d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1"}
Jul 29 10:37:19.689136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:19.689126    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3df7d7116bc53610cb56fbc400e9053a8ecf1a16319a715536397eec5eb91e1"
Jul 29 10:37:21.700482 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:21.700448    2574 generic.go:356] "Generic (PLEG): container finished" podID="b59c1858-2841-4c89-9925-e3a65595aebb" containerID="ca70571e9616e986c36902aa881e10d3cbfe5c60f34b0547f0c4a50e00f5502a" exitCode=0
Jul 29 10:37:21.700859 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:21.700503    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j" event={"ID":"b59c1858-2841-4c89-9925-e3a65595aebb","Type":"ContainerDied","Data":"ca70571e9616e986c36902aa881e10d3cbfe5c60f34b0547f0c4a50e00f5502a"}
Jul 29 10:37:22.707391 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:22.707351    2574 generic.go:356] "Generic (PLEG): container finished" podID="b59c1858-2841-4c89-9925-e3a65595aebb" containerID="4c09d6a12accd93ba4a366b18e6c736f209f107ee4363cf77ad5257ca46af4b2" exitCode=0
Jul 29 10:37:22.707778 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:22.707443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j" event={"ID":"b59c1858-2841-4c89-9925-e3a65595aebb","Type":"ContainerDied","Data":"4c09d6a12accd93ba4a366b18e6c736f209f107ee4363cf77ad5257ca46af4b2"}
Jul 29 10:37:23.714347 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.714258    2574 generic.go:356] "Generic (PLEG): container finished" podID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerID="77db1aaf4e1bba62cb1d0cabb00620618a649b70fbb2572341c0c60742571ea6" exitCode=0
Jul 29 10:37:23.714347 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.714337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws" event={"ID":"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712","Type":"ContainerDied","Data":"77db1aaf4e1bba62cb1d0cabb00620618a649b70fbb2572341c0c60742571ea6"}
Jul 29 10:37:23.857458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.857432    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:23.966376 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.966278    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmpfc\" (UniqueName: \"kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc\") pod \"b59c1858-2841-4c89-9925-e3a65595aebb\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") "
Jul 29 10:37:23.966376 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.966321    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util\") pod \"b59c1858-2841-4c89-9925-e3a65595aebb\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") "
Jul 29 10:37:23.966376 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.966378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle\") pod \"b59c1858-2841-4c89-9925-e3a65595aebb\" (UID: \"b59c1858-2841-4c89-9925-e3a65595aebb\") "
Jul 29 10:37:23.966885 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.966858    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle" (OuterVolumeSpecName: "bundle") pod "b59c1858-2841-4c89-9925-e3a65595aebb" (UID: "b59c1858-2841-4c89-9925-e3a65595aebb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:23.968683 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.968663    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc" (OuterVolumeSpecName: "kube-api-access-kmpfc") pod "b59c1858-2841-4c89-9925-e3a65595aebb" (UID: "b59c1858-2841-4c89-9925-e3a65595aebb"). InnerVolumeSpecName "kube-api-access-kmpfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:37:23.970799 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:23.970777    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util" (OuterVolumeSpecName: "util") pod "b59c1858-2841-4c89-9925-e3a65595aebb" (UID: "b59c1858-2841-4c89-9925-e3a65595aebb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:24.067350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.067312    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:24.067350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.067343    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmpfc\" (UniqueName: \"kubernetes.io/projected/b59c1858-2841-4c89-9925-e3a65595aebb-kube-api-access-kmpfc\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:24.067350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.067354    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b59c1858-2841-4c89-9925-e3a65595aebb-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:24.721757 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.721657    2574 generic.go:356] "Generic (PLEG): container finished" podID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerID="b980ceb294cfe178cc550b69a56897c8d997b5b3ccf1402c39eb4366eb24b22b" exitCode=0
Jul 29 10:37:24.721757 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.721746    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws" event={"ID":"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712","Type":"ContainerDied","Data":"b980ceb294cfe178cc550b69a56897c8d997b5b3ccf1402c39eb4366eb24b22b"}
Jul 29 10:37:24.723751 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.723726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j" event={"ID":"b59c1858-2841-4c89-9925-e3a65595aebb","Type":"ContainerDied","Data":"2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7"}
Jul 29 10:37:24.723901 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.723762    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b1c95e98356736615ca3a975724886ecb121dd351aa0fc7db6b681ec6a762f7"
Jul 29 10:37:24.723901 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:24.723777    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0vhk4j"
Jul 29 10:37:25.862635 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.862610    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:25.882382 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.882282    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bftxl\" (UniqueName: \"kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl\") pod \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") "
Jul 29 10:37:25.882382 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.882337    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util\") pod \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") "
Jul 29 10:37:25.882612 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.882461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle\") pod \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\" (UID: \"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712\") "
Jul 29 10:37:25.883656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.883624    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle" (OuterVolumeSpecName: "bundle") pod "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" (UID: "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:25.884917 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.884891    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl" (OuterVolumeSpecName: "kube-api-access-bftxl") pod "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" (UID: "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712"). InnerVolumeSpecName "kube-api-access-bftxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:37:25.889283 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.889249    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util" (OuterVolumeSpecName: "util") pod "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" (UID: "a96b743c-df3c-46a1-8a3a-7fcdfe5c2712"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:37:25.983572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.983472    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-bundle\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:25.983572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.983516    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bftxl\" (UniqueName: \"kubernetes.io/projected/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-kube-api-access-bftxl\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:25.983572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:25.983526    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a96b743c-df3c-46a1-8a3a-7fcdfe5c2712-util\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:37:26.740772 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:26.740688    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws" event={"ID":"a96b743c-df3c-46a1-8a3a-7fcdfe5c2712","Type":"ContainerDied","Data":"d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21"}
Jul 29 10:37:26.740772 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:26.740708    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053jh5ws"
Jul 29 10:37:26.740772 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:26.740758    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07a8e3a586803ec4e2e3fb34ede00211142043349e88a9b396b9bbac6922d21"
Jul 29 10:37:44.949653 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.949555    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"]
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950010    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950030    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950048    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950057    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950067    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950075    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950088    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950096    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950105    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950112    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950122    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950130    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950144    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950152    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950161    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950169    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950187    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950195    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950205    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950233    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="util"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950242    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950251    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="pull"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950259    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="extract"
Jul 29 10:37:44.950284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950266    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="extract"
Jul 29 10:37:44.951676 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950356    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="714748e4-fdc8-4325-8063-cded5b69c6c8" containerName="extract"
Jul 29 10:37:44.951676 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950368    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b59c1858-2841-4c89-9925-e3a65595aebb" containerName="extract"
Jul 29 10:37:44.951676 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950382    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a96b743c-df3c-46a1-8a3a-7fcdfe5c2712" containerName="extract"
Jul 29 10:37:44.951676 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.950393    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47c56db-97b1-429c-bee2-2d24cf253186" containerName="extract"
Jul 29 10:37:44.953560 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.953540    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:44.955893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.955872    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 10:37:44.956097 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.956068    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 10:37:44.956097 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.956077    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 10:37:44.956277 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.956106    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4wxg\""
Jul 29 10:37:44.956277 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.956198    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 10:37:44.961158 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:44.961134    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"]
Jul 29 10:37:45.051632 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.051587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ac9ee07-648c-493f-81e2-585817064060-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.051813 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.051653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.051813 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.051729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6kn\" (UniqueName: \"kubernetes.io/projected/0ac9ee07-648c-493f-81e2-585817064060-kube-api-access-tp6kn\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.152999 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.152952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ac9ee07-648c-493f-81e2-585817064060-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.153172 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.153033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.153172 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.153079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6kn\" (UniqueName: \"kubernetes.io/projected/0ac9ee07-648c-493f-81e2-585817064060-kube-api-access-tp6kn\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.153305 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:37:45.153189    2574 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 29 10:37:45.153305 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:37:45.153294    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert podName:0ac9ee07-648c-493f-81e2-585817064060 nodeName:}" failed. No retries permitted until 2026-07-29 10:37:45.653274381 +0000 UTC m=+541.833656241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-gwjp6" (UID: "0ac9ee07-648c-493f-81e2-585817064060") : secret "plugin-serving-cert" not found
Jul 29 10:37:45.153579 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.153554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0ac9ee07-648c-493f-81e2-585817064060-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.162350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.162326    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6kn\" (UniqueName: \"kubernetes.io/projected/0ac9ee07-648c-493f-81e2-585817064060-kube-api-access-tp6kn\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.657101 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.657042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.659749 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.659723    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ac9ee07-648c-493f-81e2-585817064060-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-gwjp6\" (UID: \"0ac9ee07-648c-493f-81e2-585817064060\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:45.863300 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:45.863266    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"
Jul 29 10:37:46.002978 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:46.002944    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6"]
Jul 29 10:37:46.007689 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:37:46.007658    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ac9ee07_648c_493f_81e2_585817064060.slice/crio-00136a21bcec13d945c1ce409c664e2895344fd384ee6c0f5444a2a26afc94be WatchSource:0}: Error finding container 00136a21bcec13d945c1ce409c664e2895344fd384ee6c0f5444a2a26afc94be: Status 404 returned error can't find the container with id 00136a21bcec13d945c1ce409c664e2895344fd384ee6c0f5444a2a26afc94be
Jul 29 10:37:46.843796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:46.843728    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6" event={"ID":"0ac9ee07-648c-493f-81e2-585817064060","Type":"ContainerStarted","Data":"00136a21bcec13d945c1ce409c664e2895344fd384ee6c0f5444a2a26afc94be"}
Jul 29 10:37:50.868742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:50.868652    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6" event={"ID":"0ac9ee07-648c-493f-81e2-585817064060","Type":"ContainerStarted","Data":"43b757e76c971ac36c7f17d6aa5326e7ff1a4d2e9babe0fdf6d999f63be126ce"}
Jul 29 10:37:50.883856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:37:50.883807    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-gwjp6" podStartSLOduration=2.35054577 podStartE2EDuration="6.883791065s" totalImagesPullingTime="4.533245295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:37:50.882604444 +0000 UTC m=+547.062986317" watchObservedRunningTime="2026-07-29 10:37:50.883791065 +0000 UTC m=+547.064172969"
Jul 29 10:38:27.162255 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.162202    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"]
Jul 29 10:38:27.182040 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.182010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"]
Jul 29 10:38:27.182233 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.182136    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.184267 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.184248    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 10:38:27.187731 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.187712    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"]
Jul 29 10:38:27.278945 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.278893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8099bc5-4964-415e-bd97-b807216c74ab-config-file\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.278945 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.278936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj9m\" (UniqueName: \"kubernetes.io/projected/d8099bc5-4964-415e-bd97-b807216c74ab-kube-api-access-sfj9m\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.379721 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.379671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8099bc5-4964-415e-bd97-b807216c74ab-config-file\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.379721 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.379721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj9m\" (UniqueName: \"kubernetes.io/projected/d8099bc5-4964-415e-bd97-b807216c74ab-kube-api-access-sfj9m\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.380464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.380443    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d8099bc5-4964-415e-bd97-b807216c74ab-config-file\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.389107 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.389087    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj9m\" (UniqueName: \"kubernetes.io/projected/d8099bc5-4964-415e-bd97-b807216c74ab-kube-api-access-sfj9m\") pod \"limitador-limitador-85d9db4d5d-c9rqr\" (UID: \"d8099bc5-4964-415e-bd97-b807216c74ab\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.494000 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.493911    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:27.640596 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.640571    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"]
Jul 29 10:38:27.643143 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:38:27.643103    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8099bc5_4964_415e_bd97_b807216c74ab.slice/crio-b614c4d2e05ee46a9feff7bcb307ccb8961387a0c9687956e889dfea84c3765c WatchSource:0}: Error finding container b614c4d2e05ee46a9feff7bcb307ccb8961387a0c9687956e889dfea84c3765c: Status 404 returned error can't find the container with id b614c4d2e05ee46a9feff7bcb307ccb8961387a0c9687956e889dfea84c3765c
Jul 29 10:38:27.710796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.710764    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:27.715463 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.715442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:27.717507 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.717485    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x8ngz\""
Jul 29 10:38:27.720983 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.720958    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:27.782480 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.782438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsv2x\" (UniqueName: \"kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x\") pod \"authorino-c548b5d7d-j8684\" (UID: \"e8f11fcd-4716-4d78-8a65-7cff774387a4\") " pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:27.882866 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.882832    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsv2x\" (UniqueName: \"kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x\") pod \"authorino-c548b5d7d-j8684\" (UID: \"e8f11fcd-4716-4d78-8a65-7cff774387a4\") " pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:27.891451 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:27.891418    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsv2x\" (UniqueName: \"kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x\") pod \"authorino-c548b5d7d-j8684\" (UID: \"e8f11fcd-4716-4d78-8a65-7cff774387a4\") " pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:28.026084 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:28.026038    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:28.052261 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:28.052186    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr" event={"ID":"d8099bc5-4964-415e-bd97-b807216c74ab","Type":"ContainerStarted","Data":"b614c4d2e05ee46a9feff7bcb307ccb8961387a0c9687956e889dfea84c3765c"}
Jul 29 10:38:28.167839 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:28.167804    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:28.172017 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:38:28.171989    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8f11fcd_4716_4d78_8a65_7cff774387a4.slice/crio-b090f78fa28dcf5e25ceb9c9f748f3f4a13915f6155f208e189ab5a51e08d71d WatchSource:0}: Error finding container b090f78fa28dcf5e25ceb9c9f748f3f4a13915f6155f208e189ab5a51e08d71d: Status 404 returned error can't find the container with id b090f78fa28dcf5e25ceb9c9f748f3f4a13915f6155f208e189ab5a51e08d71d
Jul 29 10:38:29.063167 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:29.063124    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-j8684" event={"ID":"e8f11fcd-4716-4d78-8a65-7cff774387a4","Type":"ContainerStarted","Data":"b090f78fa28dcf5e25ceb9c9f748f3f4a13915f6155f208e189ab5a51e08d71d"}
Jul 29 10:38:31.077264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:31.077182    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr" event={"ID":"d8099bc5-4964-415e-bd97-b807216c74ab","Type":"ContainerStarted","Data":"1da9008b9cfe0bdb24b0ff181c59d04b7b5a0fe59cc81fda48ae54867f7c04be"}
Jul 29 10:38:31.077774 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:31.077395    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:31.078802 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:31.078778    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-j8684" event={"ID":"e8f11fcd-4716-4d78-8a65-7cff774387a4","Type":"ContainerStarted","Data":"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"}
Jul 29 10:38:31.094071 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:31.094021    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr" podStartSLOduration=0.866577261 podStartE2EDuration="4.094003219s" totalImagesPullingTime="3.227425958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:31.092915529 +0000 UTC m=+587.273297409" watchObservedRunningTime="2026-07-29 10:38:31.094003219 +0000 UTC m=+587.274385098"
Jul 29 10:38:31.107031 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:31.106969    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-j8684" podStartSLOduration=1.401791302 podStartE2EDuration="4.10694956s" totalImagesPullingTime="2.705158258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:31.106165631 +0000 UTC m=+587.286547521" watchObservedRunningTime="2026-07-29 10:38:31.10694956 +0000 UTC m=+587.287331458"
Jul 29 10:38:42.085586 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:42.085538    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c9rqr"
Jul 29 10:38:45.938978 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:45.938860    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:38:45.950877 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:45.939931    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:38:51.927132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.927097    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5j7sb"]
Jul 29 10:38:51.933890 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.933864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:51.936146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.936117    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 29 10:38:51.936546 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.936527    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5j7sb"]
Jul 29 10:38:51.998341 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.998307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/543a1ddc-23da-4669-9cb2-4160f91d3982-tls-cert\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:51.998341 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:51.998350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh8v9\" (UniqueName: \"kubernetes.io/projected/543a1ddc-23da-4669-9cb2-4160f91d3982-kube-api-access-bh8v9\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.099583 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.099543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/543a1ddc-23da-4669-9cb2-4160f91d3982-tls-cert\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.099759 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.099589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh8v9\" (UniqueName: \"kubernetes.io/projected/543a1ddc-23da-4669-9cb2-4160f91d3982-kube-api-access-bh8v9\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.102246 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.102203    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/543a1ddc-23da-4669-9cb2-4160f91d3982-tls-cert\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.108068 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.108042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh8v9\" (UniqueName: \"kubernetes.io/projected/543a1ddc-23da-4669-9cb2-4160f91d3982-kube-api-access-bh8v9\") pod \"authorino-7cf4765b85-5j7sb\" (UID: \"543a1ddc-23da-4669-9cb2-4160f91d3982\") " pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.244447 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.244359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-5j7sb"
Jul 29 10:38:52.384038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.384012    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-5j7sb"]
Jul 29 10:38:52.386093 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:38:52.386067    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod543a1ddc_23da_4669_9cb2_4160f91d3982.slice/crio-5f3bb80a583d8361605295d3013565a2bd340709bad81eb017a6fb0770571a10 WatchSource:0}: Error finding container 5f3bb80a583d8361605295d3013565a2bd340709bad81eb017a6fb0770571a10: Status 404 returned error can't find the container with id 5f3bb80a583d8361605295d3013565a2bd340709bad81eb017a6fb0770571a10
Jul 29 10:38:52.387517 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:52.387500    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:38:53.188019 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.187983    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5j7sb" event={"ID":"543a1ddc-23da-4669-9cb2-4160f91d3982","Type":"ContainerStarted","Data":"16b25f5569ee6765c3516c0a99c7e581d4cc762d995c2618badd47257faa85d0"}
Jul 29 10:38:53.188019 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.188025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-5j7sb" event={"ID":"543a1ddc-23da-4669-9cb2-4160f91d3982","Type":"ContainerStarted","Data":"5f3bb80a583d8361605295d3013565a2bd340709bad81eb017a6fb0770571a10"}
Jul 29 10:38:53.204545 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.204492    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-5j7sb" podStartSLOduration=1.577889561 podStartE2EDuration="2.204477041s" totalImagesPullingTime="626.58748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:38:53.202364593 +0000 UTC m=+609.382746485" watchObservedRunningTime="2026-07-29 10:38:53.204477041 +0000 UTC m=+609.384858920"
Jul 29 10:38:53.228812 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.228781    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:53.228965 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.228950    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-j8684" podUID="e8f11fcd-4716-4d78-8a65-7cff774387a4" containerName="authorino" containerID="cri-o://618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507" gracePeriod=30
Jul 29 10:38:53.496509 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.496470    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:53.612050 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.611954    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsv2x\" (UniqueName: \"kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x\") pod \"e8f11fcd-4716-4d78-8a65-7cff774387a4\" (UID: \"e8f11fcd-4716-4d78-8a65-7cff774387a4\") "
Jul 29 10:38:53.614464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.614423    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x" (OuterVolumeSpecName: "kube-api-access-jsv2x") pod "e8f11fcd-4716-4d78-8a65-7cff774387a4" (UID: "e8f11fcd-4716-4d78-8a65-7cff774387a4"). InnerVolumeSpecName "kube-api-access-jsv2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:38:53.712989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:53.712949    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsv2x\" (UniqueName: \"kubernetes.io/projected/e8f11fcd-4716-4d78-8a65-7cff774387a4-kube-api-access-jsv2x\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:38:54.194037 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.193998    2574 generic.go:356] "Generic (PLEG): container finished" podID="e8f11fcd-4716-4d78-8a65-7cff774387a4" containerID="618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507" exitCode=0
Jul 29 10:38:54.194037 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.194042    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-j8684"
Jul 29 10:38:54.194625 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.194085    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-j8684" event={"ID":"e8f11fcd-4716-4d78-8a65-7cff774387a4","Type":"ContainerDied","Data":"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"}
Jul 29 10:38:54.194625 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.194125    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-j8684" event={"ID":"e8f11fcd-4716-4d78-8a65-7cff774387a4","Type":"ContainerDied","Data":"b090f78fa28dcf5e25ceb9c9f748f3f4a13915f6155f208e189ab5a51e08d71d"}
Jul 29 10:38:54.194625 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.194142    2574 scope.go:117] "RemoveContainer" containerID="618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"
Jul 29 10:38:54.209248 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.209199    2574 scope.go:117] "RemoveContainer" containerID="618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"
Jul 29 10:38:54.209815 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:38:54.209792    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507\": container with ID starting with 618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507 not found: ID does not exist" containerID="618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"
Jul 29 10:38:54.209917 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.209824    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507"} err="failed to get container status \"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507\": rpc error: code = NotFound desc = could not find container \"618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507\": container with ID starting with 618aaf2bdf049b140e35ccd5c3958c457e612c4e57c5f1aaf73c9e6a2a7ae507 not found: ID does not exist"
Jul 29 10:38:54.223876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.223853    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:54.229953 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.229929    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-j8684"]
Jul 29 10:38:54.341779 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:38:54.341752    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f11fcd-4716-4d78-8a65-7cff774387a4" path="/var/lib/kubelet/pods/e8f11fcd-4716-4d78-8a65-7cff774387a4/volumes"
Jul 29 10:39:53.669848 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.669811    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-bg8rw"]
Jul 29 10:39:53.670296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.670135    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f11fcd-4716-4d78-8a65-7cff774387a4" containerName="authorino"
Jul 29 10:39:53.670296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.670146    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f11fcd-4716-4d78-8a65-7cff774387a4" containerName="authorino"
Jul 29 10:39:53.670296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.670230    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f11fcd-4716-4d78-8a65-7cff774387a4" containerName="authorino"
Jul 29 10:39:53.674010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.673995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.675990 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.675969    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 29 10:39:53.676128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.675969    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dqz2x\""
Jul 29 10:39:53.680319 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.680293    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bg8rw"]
Jul 29 10:39:53.836328 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.836290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.836505 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.836342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d864\" (UniqueName: \"kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.937212 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.937130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.937212 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.937177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d864\" (UniqueName: \"kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.937538 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.937508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.945673 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.945644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d864\" (UniqueName: \"kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864\") pod \"s3-init-bg8rw\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") " pod="kserve/s3-init-bg8rw"
Jul 29 10:39:53.984659 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:53.984609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bg8rw"
Jul 29 10:39:54.125973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:54.125825    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-bg8rw"]
Jul 29 10:39:54.128345 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:39:54.128316    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd3919b7_dcc8_43b8_b2c1_94c6fcb8653e.slice/crio-cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458 WatchSource:0}: Error finding container cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458: Status 404 returned error can't find the container with id cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458
Jul 29 10:39:54.512713 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:54.512678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerStarted","Data":"cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458"}
Jul 29 10:39:58.536391 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:39:58.536352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerStarted","Data":"f28fc7faa48f2c013f6d89c755c76ddd56dade2433336d5241d892f5f766b7b0"}
Jul 29 10:40:00.548369 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:40:00.548332    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerID="f28fc7faa48f2c013f6d89c755c76ddd56dade2433336d5241d892f5f766b7b0" exitCode=0
Jul 29 10:40:00.548798 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:40:00.548406    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerDied","Data":"f28fc7faa48f2c013f6d89c755c76ddd56dade2433336d5241d892f5f766b7b0"}
Jul 29 10:40:05.583062 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:40:05.583026    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerStarted","Data":"51786479b22186af2ae8d46e89b8961873eddadc4dd35333735d7b75b81b42e4"}
Jul 29 10:40:05.600140 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:40:05.600087    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-bg8rw" podStartSLOduration=2.432969701 podStartE2EDuration="12.600068968s" totalImagesPullingTime="8.068492497s" totalInitContainerRuntime="2.09860677s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:39:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:40:05.597344507 +0000 UTC m=+681.777726386" watchObservedRunningTime="2026-07-29 10:40:05.600068968 +0000 UTC m=+681.780450887"
Jul 29 10:41:10.906874 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:10.906835    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerID="51786479b22186af2ae8d46e89b8961873eddadc4dd35333735d7b75b81b42e4" exitCode=0
Jul 29 10:41:10.907350 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:10.906894    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerDied","Data":"51786479b22186af2ae8d46e89b8961873eddadc4dd35333735d7b75b81b42e4"}
Jul 29 10:41:12.047156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.047132    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bg8rw"
Jul 29 10:41:12.124479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.124441    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache\") pod \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") "
Jul 29 10:41:12.124650 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.124487    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d864\" (UniqueName: \"kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864\") pod \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\" (UID: \"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e\") "
Jul 29 10:41:12.127084 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.127051    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864" (OuterVolumeSpecName: "kube-api-access-4d864") pod "dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" (UID: "dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e"). InnerVolumeSpecName "kube-api-access-4d864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:41:12.143844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.143807    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" (UID: "dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:41:12.225974 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.225884    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:41:12.225974 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.225916    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d864\" (UniqueName: \"kubernetes.io/projected/dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e-kube-api-access-4d864\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:41:12.918232 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.918183    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-bg8rw" event={"ID":"dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e","Type":"ContainerDied","Data":"cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458"}
Jul 29 10:41:12.918232 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.918203    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-bg8rw"
Jul 29 10:41:12.918443 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:12.918248    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf4b9a4cd930fd9ed5d96026dac12ca3eaa4956b30b4900a9555305c2a68e458"
Jul 29 10:41:40.221279 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221245    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:41:40.221688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221603    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerName="download-hf-model"
Jul 29 10:41:40.221688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221615    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerName="download-hf-model"
Jul 29 10:41:40.221688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221633    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerName="s3-init"
Jul 29 10:41:40.221688 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221638    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerName="s3-init"
Jul 29 10:41:40.221818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.221699    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e" containerName="s3-init"
Jul 29 10:41:40.224785 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.224768    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.227005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.226982    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-6vtk4\""
Jul 29 10:41:40.227135 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.227038    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Jul 29 10:41:40.227135 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.227060    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Jul 29 10:41:40.227135 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.227126    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 29 10:41:40.227327 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.227312    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Jul 29 10:41:40.235373 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.235352    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:41:40.271365 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zw2\" (UniqueName: \"kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271638 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271638 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.271702 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.271647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372292 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zw2\" (UniqueName: \"kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372860 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372919 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.372979 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.372938    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.373041 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.373019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.374819 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.374791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.375055 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.375037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.379445 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.379422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zw2\" (UniqueName: \"kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.535714 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.535670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:41:40.696734 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:40.696707    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:41:40.699425 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:41:40.699389    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7d67ae_2430_45e0_809b_9fef7d326df8.slice/crio-819a8ae02ac94a3959a0700325308bfde3e862c556592cc736e9f36bbb6c047b WatchSource:0}: Error finding container 819a8ae02ac94a3959a0700325308bfde3e862c556592cc736e9f36bbb6c047b: Status 404 returned error can't find the container with id 819a8ae02ac94a3959a0700325308bfde3e862c556592cc736e9f36bbb6c047b
Jul 29 10:41:41.064008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:41.063965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerStarted","Data":"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"}
Jul 29 10:41:41.064205 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:41.064036    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerStarted","Data":"819a8ae02ac94a3959a0700325308bfde3e862c556592cc736e9f36bbb6c047b"}
Jul 29 10:41:46.092590 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:46.092555    2574 generic.go:356] "Generic (PLEG): container finished" podID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerID="b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9" exitCode=0
Jul 29 10:41:46.092964 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:41:46.092619    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerDied","Data":"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"}
Jul 29 10:42:16.289588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:16.289554    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerStarted","Data":"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"}
Jul 29 10:42:16.314830 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:16.314760    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podStartSLOduration=1.885828228 podStartE2EDuration="36.314739325s" totalImagesPullingTime="29.642357712s" totalInitContainerRuntime="4.786553385s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:42:16.312335872 +0000 UTC m=+812.492717753" watchObservedRunningTime="2026-07-29 10:42:16.314739325 +0000 UTC m=+812.495121205"
Jul 29 10:42:16.536507 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:16.536468    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:42:20.535870 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:20.535826    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:42:20.537408 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:20.537379    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:42:30.536902 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:30.536845    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:42:40.536696 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:40.536647    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:42:50.537133 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:42:50.537082    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:00.536607 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:00.536565    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:10.536900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:10.536852    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:20.537089 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:20.537042    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:30.536203 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:30.536148    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:40.536384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:40.536332    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:43:45.979417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:45.979316    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:43:45.979417 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:45.979324    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:43:50.536628 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:43:50.536583    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:44:00.536286 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:00.536242    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:44:10.536750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:10.536705    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:44:20.546480 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:20.546444    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:44:20.554532 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:20.554504    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:44:43.024781 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:43.024742    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:44:58.048005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.047955    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" containerID="cri-o://ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78" gracePeriod=45
Jul 29 10:44:58.548127 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.548077    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jul 29 10:44:58.806789 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.806731    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:44:58.873744 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873718    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.873900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873756    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.873900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.873900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873833    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.873900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873853    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.873900 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873888    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.874146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.873913    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6zw2\" (UniqueName: \"kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2\") pod \"df7d67ae-2430-45e0-809b-9fef7d326df8\" (UID: \"df7d67ae-2430-45e0-809b-9fef7d326df8\") "
Jul 29 10:44:58.874146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.874129    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache" (OuterVolumeSpecName: "model-cache") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:58.874535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.874510    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home" (OuterVolumeSpecName: "home") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:58.876405 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.876377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:44:58.876506 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.876448    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm" (OuterVolumeSpecName: "dshm") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:58.876650 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.876628    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2" (OuterVolumeSpecName: "kube-api-access-f6zw2") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "kube-api-access-f6zw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:44:58.886479 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.886449    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:58.930478 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.930434    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df7d67ae-2430-45e0-809b-9fef7d326df8" (UID: "df7d67ae-2430-45e0-809b-9fef7d326df8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:44:58.974740 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974704    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974740 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974734    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974740 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974744    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974757    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zw2\" (UniqueName: \"kubernetes.io/projected/df7d67ae-2430-45e0-809b-9fef7d326df8-kube-api-access-f6zw2\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974769    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7d67ae-2430-45e0-809b-9fef7d326df8-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974777    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:58.974973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:58.974785    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7d67ae-2430-45e0-809b-9fef7d326df8-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:44:59.167146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.167062    2574 generic.go:356] "Generic (PLEG): container finished" podID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerID="ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78" exitCode=0
Jul 29 10:44:59.167146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.167109    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"
Jul 29 10:44:59.167678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.167110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerDied","Data":"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"}
Jul 29 10:44:59.167678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.167280    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57" event={"ID":"df7d67ae-2430-45e0-809b-9fef7d326df8","Type":"ContainerDied","Data":"819a8ae02ac94a3959a0700325308bfde3e862c556592cc736e9f36bbb6c047b"}
Jul 29 10:44:59.167678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.167324    2574 scope.go:117] "RemoveContainer" containerID="ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"
Jul 29 10:44:59.177058 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.177038    2574 scope.go:117] "RemoveContainer" containerID="b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"
Jul 29 10:44:59.185498 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.185483    2574 scope.go:117] "RemoveContainer" containerID="ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"
Jul 29 10:44:59.185793 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:44:59.185769    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78\": container with ID starting with ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78 not found: ID does not exist" containerID="ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"
Jul 29 10:44:59.185882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.185803    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78"} err="failed to get container status \"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78\": rpc error: code = NotFound desc = could not find container \"ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78\": container with ID starting with ee1d03cb9754df4d4a4015c570305c05354290707535fe1e3a6b3020b99b7e78 not found: ID does not exist"
Jul 29 10:44:59.185882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.185828    2574 scope.go:117] "RemoveContainer" containerID="b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"
Jul 29 10:44:59.186187 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:44:59.186159    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9\": container with ID starting with b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9 not found: ID does not exist" containerID="b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"
Jul 29 10:44:59.186258 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.186200    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9"} err="failed to get container status \"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9\": rpc error: code = NotFound desc = could not find container \"b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9\": container with ID starting with b1a8535d9aa158dadc705f993c6cd809c0b13f4ae4099bc5fc72101427cddcf9 not found: ID does not exist"
Jul 29 10:44:59.188841 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.188823    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:44:59.194823 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:44:59.194801    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-764c46744cwlp57"]
Jul 29 10:45:00.342728 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:00.342689    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" path="/var/lib/kubelet/pods/df7d67ae-2430-45e0-809b-9fef7d326df8/volumes"
Jul 29 10:45:11.659683 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.659576    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:45:11.660125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.659977    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="storage-initializer"
Jul 29 10:45:11.660125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.659992    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="storage-initializer"
Jul 29 10:45:11.660125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.660007    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main"
Jul 29 10:45:11.660125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.660012    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main"
Jul 29 10:45:11.660125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.660074    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df7d67ae-2430-45e0-809b-9fef7d326df8" containerName="main"
Jul 29 10:45:11.663386 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.663344    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.666808 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-5v7b2\""
Jul 29 10:45:11.666808 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666792    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 29 10:45:11.666808 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666771    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 29 10:45:11.666808 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 29 10:45:11.667264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-hk947\""
Jul 29 10:45:11.667264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.666771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 29 10:45:11.679974 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.679935    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:45:11.783932 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.783895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.783944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.783978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.784036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.784099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784132 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.784123    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm54b\" (UniqueName: \"kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.784348 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.784145    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm54b\" (UniqueName: \"kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884695    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.884924 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.884805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.885189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.885051    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.885189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.885080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.885189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.885161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.885189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.885175    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.887330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.887300    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.887536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.887516    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.892414 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.892391    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm54b\" (UniqueName: \"kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:11.979733 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:11.979626    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:12.131397 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:12.131358    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:45:12.133053 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:45:12.133026    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd92a51_cd3e_4878_8183_a6147ff9754b.slice/crio-53f9b10e9228145cbd8f187966012766e4618b14ef5bd6ae2186d17e19ec67aa WatchSource:0}: Error finding container 53f9b10e9228145cbd8f187966012766e4618b14ef5bd6ae2186d17e19ec67aa: Status 404 returned error can't find the container with id 53f9b10e9228145cbd8f187966012766e4618b14ef5bd6ae2186d17e19ec67aa
Jul 29 10:45:12.235254 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:12.235137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerStarted","Data":"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"}
Jul 29 10:45:12.235254 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:12.235201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerStarted","Data":"53f9b10e9228145cbd8f187966012766e4618b14ef5bd6ae2186d17e19ec67aa"}
Jul 29 10:45:17.264023 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:17.263988    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerID="4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4" exitCode=0
Jul 29 10:45:17.264463 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:17.264061    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerDied","Data":"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"}
Jul 29 10:45:18.272271 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:18.272207    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerStarted","Data":"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"}
Jul 29 10:45:18.294872 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:18.294819    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.106652575 podStartE2EDuration="7.294803272s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.188150697s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:45:18.291802079 +0000 UTC m=+994.472183972" watchObservedRunningTime="2026-07-29 10:45:18.294803272 +0000 UTC m=+994.475185152"
Jul 29 10:45:18.980187 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:18.980150    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:21.980121 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:21.980083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:45:21.981670 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:21.981640    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:45:31.981060 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:31.981015    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:45:41.980125 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:41.980056    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:45:51.980895 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:45:51.980850    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:01.980656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:01.980599    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:11.980572 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:11.980523    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:21.980940 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:21.980888    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:31.980936 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:31.980892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:41.980380 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:41.980285    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:46:51.980336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:46:51.980283    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jul 29 10:47:01.990491 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:01.990458    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:47:01.998119 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:01.998086    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:47:31.118087 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:31.118045    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:47:46.141374 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:46.141312    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main" containerID="cri-o://77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3" gracePeriod=45
Jul 29 10:47:46.907121 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:46.907090    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:47:47.050330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050296    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm54b\" (UniqueName: \"kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050419    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050471    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.050794 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home\") pod \"cdd92a51-cd3e-4878-8183-a6147ff9754b\" (UID: \"cdd92a51-cd3e-4878-8183-a6147ff9754b\") "
Jul 29 10:47:47.051726 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.050747    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:47.051726 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.051276    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home" (OuterVolumeSpecName: "home") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:47.051726 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.051679    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.051726 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.051697    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.053026 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.052982    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm" (OuterVolumeSpecName: "dshm") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:47.053234 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.053183    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b" (OuterVolumeSpecName: "kube-api-access-wm54b") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "kube-api-access-wm54b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:47:47.053306 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.053194    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:47:47.064301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.064251    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:47.085304 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.085268    2574 generic.go:356] "Generic (PLEG): container finished" podID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerID="77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3" exitCode=0
Jul 29 10:47:47.085304 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.085308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerDied","Data":"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"}
Jul 29 10:47:47.085528 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.085323    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 29 10:47:47.085528 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.085352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"cdd92a51-cd3e-4878-8183-a6147ff9754b","Type":"ContainerDied","Data":"53f9b10e9228145cbd8f187966012766e4618b14ef5bd6ae2186d17e19ec67aa"}
Jul 29 10:47:47.085528 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.085375    2574 scope.go:117] "RemoveContainer" containerID="77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"
Jul 29 10:47:47.094390 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.094371    2574 scope.go:117] "RemoveContainer" containerID="4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"
Jul 29 10:47:47.104325 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.104300    2574 scope.go:117] "RemoveContainer" containerID="77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"
Jul 29 10:47:47.104663 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:47:47.104637    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3\": container with ID starting with 77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3 not found: ID does not exist" containerID="77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"
Jul 29 10:47:47.104733 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.104679    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3"} err="failed to get container status \"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3\": rpc error: code = NotFound desc = could not find container \"77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3\": container with ID starting with 77fd0ca1bdb2856df66f66fbdfcc07cfffea24b90628b2f9592871aba2f029c3 not found: ID does not exist"
Jul 29 10:47:47.104733 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.104705    2574 scope.go:117] "RemoveContainer" containerID="4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"
Jul 29 10:47:47.104955 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:47:47.104938    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4\": container with ID starting with 4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4 not found: ID does not exist" containerID="4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"
Jul 29 10:47:47.105010 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.104969    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4"} err="failed to get container status \"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4\": rpc error: code = NotFound desc = could not find container \"4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4\": container with ID starting with 4a40afbd3c429a8c2973fbd98be8603f990a85da8ec93588de0e28300f71f3a4 not found: ID does not exist"
Jul 29 10:47:47.121964 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.121914    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdd92a51-cd3e-4878-8183-a6147ff9754b" (UID: "cdd92a51-cd3e-4878-8183-a6147ff9754b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:47:47.152464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.152409    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.152464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.152455    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdd92a51-cd3e-4878-8183-a6147ff9754b-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.152464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.152467    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.152464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.152477    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm54b\" (UniqueName: \"kubernetes.io/projected/cdd92a51-cd3e-4878-8183-a6147ff9754b-kube-api-access-wm54b\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.152464 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.152485    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdd92a51-cd3e-4878-8183-a6147ff9754b-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:47:47.409791 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.409758    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:47:47.414278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:47.414251    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 29 10:47:48.341816 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:47:48.341785    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" path="/var/lib/kubelet/pods/cdd92a51-cd3e-4878-8183-a6147ff9754b/volumes"
Jul 29 10:48:28.622958 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.622919    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:48:28.623533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.623254    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="storage-initializer"
Jul 29 10:48:28.623533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.623266    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="storage-initializer"
Jul 29 10:48:28.623533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.623282    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main"
Jul 29 10:48:28.623533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.623288    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main"
Jul 29 10:48:28.623533 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.623348    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd92a51-cd3e-4878-8183-a6147ff9754b" containerName="main"
Jul 29 10:48:28.626390 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.626371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.629007 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.628981    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-8gpj9\""
Jul 29 10:48:28.629128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.628982    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 29 10:48:28.629128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.628990    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 29 10:48:28.629128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.628994    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 29 10:48:28.636351 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.636329    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:48:28.657815 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.657783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.657994 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.657835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmfxp\" (UniqueName: \"kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.657994 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.657860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.657994 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.657893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.657994 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.657975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.658318 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.658043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.758525 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.758724 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmfxp\" (UniqueName: \"kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.758724 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.758724 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.759101 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.759101 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.758795    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.759284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.759147    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.759284 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.759166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.759593 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.759570    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.761154 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.761127    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.761292 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.761276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.768254 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.768209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmfxp\" (UniqueName: \"kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp\") pod \"scheduler-inline-config-test-kserve-8655bc757-vphqp\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:28.936946 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:28.936851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:29.086744 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:29.086714    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:48:29.089053 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:48:29.089025    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08710174_8f82_4758_a7b3_27161ea3e561.slice/crio-c821407cd5887ed7ccf8aa21c151e5d519b269b3601c48a921eecc0bbde8a0d7 WatchSource:0}: Error finding container c821407cd5887ed7ccf8aa21c151e5d519b269b3601c48a921eecc0bbde8a0d7: Status 404 returned error can't find the container with id c821407cd5887ed7ccf8aa21c151e5d519b269b3601c48a921eecc0bbde8a0d7
Jul 29 10:48:29.091026 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:29.091002    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:48:29.303297 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:29.303259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" event={"ID":"08710174-8f82-4758-a7b3-27161ea3e561","Type":"ContainerStarted","Data":"c821407cd5887ed7ccf8aa21c151e5d519b269b3601c48a921eecc0bbde8a0d7"}
Jul 29 10:48:31.316486 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:31.316445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" event={"ID":"08710174-8f82-4758-a7b3-27161ea3e561","Type":"ContainerStarted","Data":"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"}
Jul 29 10:48:31.333853 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:31.333797    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" podStartSLOduration=1.6743464179999998 podStartE2EDuration="3.333777428s" totalImagesPullingTime="1.65943101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:48:31.331317844 +0000 UTC m=+1187.511699724" watchObservedRunningTime="2026-07-29 10:48:31.333777428 +0000 UTC m=+1187.514159308"
Jul 29 10:48:31.937370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:31.937325    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:38.937563 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:38.937509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:38.950426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:38.950390    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:39.366913 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:39.366885    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:48:46.014785 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:46.014761    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:48:46.030433 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:48:46.015336    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:49:01.551724 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:01.551685    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:49:01.562570 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:01.562524    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:49:01.562570 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:01.562570 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:01.562570 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30" cmd=["/bin/sleep","15"]
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:01.562587    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" message="" err=<
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:01.562600    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" podUID="08710174-8f82-4758-a7b3-27161ea3e561" containerName="main" containerID="cri-o://01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"
Jul 29 10:49:01.562969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:01.562654    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" podUID="08710174-8f82-4758-a7b3-27161ea3e561" containerName="main" containerID="cri-o://01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30" gracePeriod=60
Jul 29 10:49:01.923982 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:01.923955    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:49:02.054911 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.054872    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.054911 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.054915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.055155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.054930    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.055155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.054957    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.055155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.054988    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.055155 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.055020    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmfxp\" (UniqueName: \"kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp\") pod \"08710174-8f82-4758-a7b3-27161ea3e561\" (UID: \"08710174-8f82-4758-a7b3-27161ea3e561\") "
Jul 29 10:49:02.055424 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.055213    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache" (OuterVolumeSpecName: "model-cache") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:02.055424 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.055250    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home" (OuterVolumeSpecName: "home") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:02.055424 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.055270    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:02.057445 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.057419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm" (OuterVolumeSpecName: "dshm") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:02.057552 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.057527    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp" (OuterVolumeSpecName: "kube-api-access-qmfxp") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "kube-api-access-qmfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:49:02.057594 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.057567    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08710174-8f82-4758-a7b3-27161ea3e561" (UID: "08710174-8f82-4758-a7b3-27161ea3e561"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155662    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155694    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155703    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155712    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08710174-8f82-4758-a7b3-27161ea3e561-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155719    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08710174-8f82-4758-a7b3-27161ea3e561-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.155750 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.155728    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmfxp\" (UniqueName: \"kubernetes.io/projected/08710174-8f82-4758-a7b3-27161ea3e561-kube-api-access-qmfxp\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:02.471002 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.470915    2574 generic.go:356] "Generic (PLEG): container finished" podID="08710174-8f82-4758-a7b3-27161ea3e561" containerID="01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30" exitCode=0
Jul 29 10:49:02.471002 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.470964    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"
Jul 29 10:49:02.471002 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.470982    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" event={"ID":"08710174-8f82-4758-a7b3-27161ea3e561","Type":"ContainerDied","Data":"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"}
Jul 29 10:49:02.471294 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.471026    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp" event={"ID":"08710174-8f82-4758-a7b3-27161ea3e561","Type":"ContainerDied","Data":"c821407cd5887ed7ccf8aa21c151e5d519b269b3601c48a921eecc0bbde8a0d7"}
Jul 29 10:49:02.471294 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.471050    2574 scope.go:117] "RemoveContainer" containerID="01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"
Jul 29 10:49:02.480163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.480145    2574 scope.go:117] "RemoveContainer" containerID="01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"
Jul 29 10:49:02.480478 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:02.480461    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30\": container with ID starting with 01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30 not found: ID does not exist" containerID="01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"
Jul 29 10:49:02.480536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.480486    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30"} err="failed to get container status \"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30\": rpc error: code = NotFound desc = could not find container \"01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30\": container with ID starting with 01221327e054ced3c2f46ed202839c6f51dfd5e83567c19c1f799391d965ed30 not found: ID does not exist"
Jul 29 10:49:02.489423 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.489399    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:49:02.493278 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:02.493253    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-vphqp"]
Jul 29 10:49:04.342003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:04.341967    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08710174-8f82-4758-a7b3-27161ea3e561" path="/var/lib/kubelet/pods/08710174-8f82-4758-a7b3-27161ea3e561/volumes"
Jul 29 10:49:23.936793 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.936759    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:49:23.937202 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.937084    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08710174-8f82-4758-a7b3-27161ea3e561" containerName="main"
Jul 29 10:49:23.937202 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.937096    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="08710174-8f82-4758-a7b3-27161ea3e561" containerName="main"
Jul 29 10:49:23.937202 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.937157    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="08710174-8f82-4758-a7b3-27161ea3e561" containerName="main"
Jul 29 10:49:23.940124 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.940108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:23.943161 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.943137    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 29 10:49:23.943161 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.943150    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 29 10:49:23.943361 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.943312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-7cjvh\""
Jul 29 10:49:23.943433 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.943412    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 29 10:49:23.950721 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:23.950694    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:49:24.046072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.046072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.046337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.046337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.046337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.046337 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.046272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9xv5\" (UniqueName: \"kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147140 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9xv5\" (UniqueName: \"kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.147389 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.147382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.148066 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.148029    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.148200 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.148020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.148200 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.148089    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.149951 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.149919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.150283 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.150252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.161179 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.161150    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9xv5\" (UniqueName: \"kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.251051 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.250944    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:24.393038 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.393010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:49:24.395031 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:49:24.395003    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f02da3_10c3_4547_b387_a62150e5bccc.slice/crio-9c376da966924395ad6e395e511937b037f9d1e6ca2acafaf6e56513e230091d WatchSource:0}: Error finding container 9c376da966924395ad6e395e511937b037f9d1e6ca2acafaf6e56513e230091d: Status 404 returned error can't find the container with id 9c376da966924395ad6e395e511937b037f9d1e6ca2acafaf6e56513e230091d
Jul 29 10:49:24.584338 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.584294    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" event={"ID":"89f02da3-10c3-4547-b387-a62150e5bccc","Type":"ContainerStarted","Data":"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"}
Jul 29 10:49:24.584535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.584367    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" event={"ID":"89f02da3-10c3-4547-b387-a62150e5bccc","Type":"ContainerStarted","Data":"9c376da966924395ad6e395e511937b037f9d1e6ca2acafaf6e56513e230091d"}
Jul 29 10:49:24.602775 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:24.602722    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" podStartSLOduration=1.6027044830000001 podStartE2EDuration="1.602704483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:49:24.60060601 +0000 UTC m=+1240.780987891" watchObservedRunningTime="2026-07-29 10:49:24.602704483 +0000 UTC m=+1240.783086362"
Jul 29 10:49:25.252109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:25.252071    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:34.251789 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:34.251753    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:34.265138 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:34.265108    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:34.646672 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:34.646643    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:58.065837 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.065795    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:49:58.077701 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:58.077659    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:49:58.077701 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:58.077701 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:58.077701 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1" cmd=["/bin/sleep","15"]
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.077719    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" message="" err=<
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:58.077733    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" containerName="main" containerID="cri-o://0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"
Jul 29 10:49:58.078032 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.077787    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" containerName="main" containerID="cri-o://0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1" gracePeriod=60
Jul 29 10:49:58.438020 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.437995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:58.550320 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550282    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550320 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550331    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550380    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550400    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9xv5\" (UniqueName: \"kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550588 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550457    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir\") pod \"89f02da3-10c3-4547-b387-a62150e5bccc\" (UID: \"89f02da3-10c3-4547-b387-a62150e5bccc\") "
Jul 29 10:49:58.550796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550685    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:58.550796 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550699    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home" (OuterVolumeSpecName: "home") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:58.550899 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.550796    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:58.552782 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.552751    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:49:58.552906 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.552805    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5" (OuterVolumeSpecName: "kube-api-access-t9xv5") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "kube-api-access-t9xv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:49:58.552906 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.552825    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm" (OuterVolumeSpecName: "dshm") pod "89f02da3-10c3-4547-b387-a62150e5bccc" (UID: "89f02da3-10c3-4547-b387-a62150e5bccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651580    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651609    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651619    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9xv5\" (UniqueName: \"kubernetes.io/projected/89f02da3-10c3-4547-b387-a62150e5bccc-kube-api-access-t9xv5\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651628    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f02da3-10c3-4547-b387-a62150e5bccc-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651638    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.651671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.651645    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f02da3-10c3-4547-b387-a62150e5bccc-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:49:58.769812 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.769766    2574 generic.go:356] "Generic (PLEG): container finished" podID="89f02da3-10c3-4547-b387-a62150e5bccc" containerID="0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1" exitCode=0
Jul 29 10:49:58.770018 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.769823    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"
Jul 29 10:49:58.770018 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.769852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" event={"ID":"89f02da3-10c3-4547-b387-a62150e5bccc","Type":"ContainerDied","Data":"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"}
Jul 29 10:49:58.770018 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.769902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn" event={"ID":"89f02da3-10c3-4547-b387-a62150e5bccc","Type":"ContainerDied","Data":"9c376da966924395ad6e395e511937b037f9d1e6ca2acafaf6e56513e230091d"}
Jul 29 10:49:58.770018 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.769928    2574 scope.go:117] "RemoveContainer" containerID="0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"
Jul 29 10:49:58.779501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.779481    2574 scope.go:117] "RemoveContainer" containerID="0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"
Jul 29 10:49:58.779785 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:49:58.779761    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1\": container with ID starting with 0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1 not found: ID does not exist" containerID="0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"
Jul 29 10:49:58.779827 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.779801    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1"} err="failed to get container status \"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1\": rpc error: code = NotFound desc = could not find container \"0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1\": container with ID starting with 0300ac0ff79bf05b47d58eb5e7445d964b01bbb1b7ffc8d86fb72cbc8de472e1 not found: ID does not exist"
Jul 29 10:49:58.792064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.792040    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:49:58.797671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:49:58.797648    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f8grvn"]
Jul 29 10:50:00.342264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:00.342231    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" path="/var/lib/kubelet/pods/89f02da3-10c3-4547-b387-a62150e5bccc/volumes"
Jul 29 10:50:26.127678 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.127644    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:50:26.128136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.127982    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" containerName="main"
Jul 29 10:50:26.128136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.127992    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" containerName="main"
Jul 29 10:50:26.128136 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.128047    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f02da3-10c3-4547-b387-a62150e5bccc" containerName="main"
Jul 29 10:50:26.130888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.130871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.133287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.133260    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 29 10:50:26.133838 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.133822    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-4dcrk\""
Jul 29 10:50:26.133921 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.133836    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 29 10:50:26.133921 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.133821    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 29 10:50:26.138890 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.138865    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:50:26.190057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.190057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.190310 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.190310 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.190310 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4gm5\" (UniqueName: \"kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.190310 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.190277    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291690 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291690 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291758 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4gm5\" (UniqueName: \"kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291806 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291914 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.291969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.291942    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.292050 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.292033    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.294042 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.294018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.294308 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.294292    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.300204 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.300182    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4gm5\" (UniqueName: \"kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.442015 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.441920    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:26.582761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.582732    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:50:26.585630 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:50:26.585600    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b0663f_2c6f_4e5d_a5c4_495fa9d88897.slice/crio-dc5102b2b6ba875ba985994218c7f3b2052ce3808a2f6f495355f8787c570e19 WatchSource:0}: Error finding container dc5102b2b6ba875ba985994218c7f3b2052ce3808a2f6f495355f8787c570e19: Status 404 returned error can't find the container with id dc5102b2b6ba875ba985994218c7f3b2052ce3808a2f6f495355f8787c570e19
Jul 29 10:50:26.910044 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.910006    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" event={"ID":"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897","Type":"ContainerStarted","Data":"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"}
Jul 29 10:50:26.910258 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.910063    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" event={"ID":"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897","Type":"ContainerStarted","Data":"dc5102b2b6ba875ba985994218c7f3b2052ce3808a2f6f495355f8787c570e19"}
Jul 29 10:50:26.927965 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:26.927914    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" podStartSLOduration=0.927894864 podStartE2EDuration="927.894864ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:50:26.927157307 +0000 UTC m=+1303.107539187" watchObservedRunningTime="2026-07-29 10:50:26.927894864 +0000 UTC m=+1303.108276745"
Jul 29 10:50:27.442241 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:27.442180    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:36.442141 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:36.442092    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:36.454769 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:36.454740    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:50:36.970494 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:50:36.970468    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:51:11.929062 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:11.928974    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:51:11.938338 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:51:11.938304    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:51:11.938338 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:51:11.938338 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:51:11.938338 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e" cmd=["/bin/sleep","15"]
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:11.938357    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" message="" err=<
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:51:11.938369    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main" containerID="cri-o://1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"
Jul 29 10:51:11.938608 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:11.938443    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main" containerID="cri-o://1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e" gracePeriod=60
Jul 29 10:51:11.960116 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:11.960079    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Jul 29 10:51:12.300884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.300859    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:51:12.471109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471010    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471065    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471098    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4gm5\" (UniqueName: \"kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471442 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471127    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471442 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471145    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471442 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471165    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home\") pod \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\" (UID: \"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897\") "
Jul 29 10:51:12.471442 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471412    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:12.471635 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471566    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:12.471635 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.471607    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home" (OuterVolumeSpecName: "home") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:12.473732 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.473700    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:51:12.473841 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.473736    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm" (OuterVolumeSpecName: "dshm") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:51:12.473841 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.473754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5" (OuterVolumeSpecName: "kube-api-access-m4gm5") pod "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" (UID: "b0b0663f-2c6f-4e5d-a5c4-495fa9d88897"). InnerVolumeSpecName "kube-api-access-m4gm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:51:12.572501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572471    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:12.572501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572497    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:12.572501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572506    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4gm5\" (UniqueName: \"kubernetes.io/projected/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-kube-api-access-m4gm5\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:12.572723 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572516    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:12.572723 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572526    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:12.572723 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:12.572533    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:51:13.148992 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.148957    2574 generic.go:356] "Generic (PLEG): container finished" podID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerID="1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e" exitCode=0
Jul 29 10:51:13.149481 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.149007    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"
Jul 29 10:51:13.149481 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.149011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" event={"ID":"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897","Type":"ContainerDied","Data":"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"}
Jul 29 10:51:13.149481 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.149121    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk" event={"ID":"b0b0663f-2c6f-4e5d-a5c4-495fa9d88897","Type":"ContainerDied","Data":"dc5102b2b6ba875ba985994218c7f3b2052ce3808a2f6f495355f8787c570e19"}
Jul 29 10:51:13.149481 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.149145    2574 scope.go:117] "RemoveContainer" containerID="1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"
Jul 29 10:51:13.159774 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.159749    2574 scope.go:117] "RemoveContainer" containerID="1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"
Jul 29 10:51:13.160117 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:51:13.160096    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e\": container with ID starting with 1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e not found: ID does not exist" containerID="1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"
Jul 29 10:51:13.160173 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.160128    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e"} err="failed to get container status \"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e\": rpc error: code = NotFound desc = could not find container \"1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e\": container with ID starting with 1766893e40866feb902868f948e01b342e702dbdf71bd0ef48972ad85ea3ab7e not found: ID does not exist"
Jul 29 10:51:13.170085 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.170055    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:51:13.174012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:13.173982    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-r5gpk"]
Jul 29 10:51:14.341854 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:14.341819    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" path="/var/lib/kubelet/pods/b0b0663f-2c6f-4e5d-a5c4-495fa9d88897/volumes"
Jul 29 10:51:36.427918 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.427876    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:51:36.429070 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.429039    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main"
Jul 29 10:51:36.429266 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.429250    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main"
Jul 29 10:51:36.429611 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.429591    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b0663f-2c6f-4e5d-a5c4-495fa9d88897" containerName="main"
Jul 29 10:51:36.438294 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.438270    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.441536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.441503    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 29 10:51:36.441706 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.441537    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 29 10:51:36.442000 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.441980    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-k6qbt\""
Jul 29 10:51:36.442355 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.442334    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 29 10:51:36.443348 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.443326    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:51:36.584886 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.584829    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.585103 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.584934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkbn\" (UniqueName: \"kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.585103 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.584962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.585103 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.584983    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.585103 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.585080    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.585314 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.585124    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.644553 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.644519    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:51:36.647969 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.647947    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.650287 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.650258    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-c8zd7\""
Jul 29 10:51:36.655406 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.655376    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:51:36.686580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.686580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686520    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.686580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.686580 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.686937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.686937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkbn\" (UniqueName: \"kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.687045 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.686993    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.687045 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.687030    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.687119 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.687053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.688908 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.688878    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.689156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.689138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.695189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.695167    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkbn\" (UniqueName: \"kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-hbk44\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.749910 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.749865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:36.787470 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.787437    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.787643 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.787493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlmtt\" (UniqueName: \"kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.888862 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.888816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlmtt\" (UniqueName: \"kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.889068 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.888935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.892201 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.892174    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.893760 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.893662    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:51:36.896151 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:51:36.896127    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200087e0_1b7c_40a7_851b_ef45b3352661.slice/crio-c28ef2bf0e97699263b7ffd4ced015f56e606ed14ac181fe4144d8b7f6bab760 WatchSource:0}: Error finding container c28ef2bf0e97699263b7ffd4ced015f56e606ed14ac181fe4144d8b7f6bab760: Status 404 returned error can't find the container with id c28ef2bf0e97699263b7ffd4ced015f56e606ed14ac181fe4144d8b7f6bab760
Jul 29 10:51:36.897882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.897862    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlmtt\" (UniqueName: \"kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:36.959393 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:36.959368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:37.103394 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.103366    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:51:37.105815 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:51:37.105787    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad094bcf_53e0_4106_a091_6602d1f66130.slice/crio-9fae34902796cc6401404d0b4131fb5ff09328dc57a59f01207746d8c94ff941 WatchSource:0}: Error finding container 9fae34902796cc6401404d0b4131fb5ff09328dc57a59f01207746d8c94ff941: Status 404 returned error can't find the container with id 9fae34902796cc6401404d0b4131fb5ff09328dc57a59f01207746d8c94ff941
Jul 29 10:51:37.270083 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.270042    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" event={"ID":"ad094bcf-53e0-4106-a091-6602d1f66130","Type":"ContainerStarted","Data":"9fae34902796cc6401404d0b4131fb5ff09328dc57a59f01207746d8c94ff941"}
Jul 29 10:51:37.271853 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.271818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" event={"ID":"200087e0-1b7c-40a7-851b-ef45b3352661","Type":"ContainerStarted","Data":"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"}
Jul 29 10:51:37.271994 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.271881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" event={"ID":"200087e0-1b7c-40a7-851b-ef45b3352661","Type":"ContainerStarted","Data":"c28ef2bf0e97699263b7ffd4ced015f56e606ed14ac181fe4144d8b7f6bab760"}
Jul 29 10:51:37.292152 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.292086    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" podStartSLOduration=1.292061114 podStartE2EDuration="1.292061114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:37.289940913 +0000 UTC m=+1373.470322797" watchObservedRunningTime="2026-07-29 10:51:37.292061114 +0000 UTC m=+1373.472442995"
Jul 29 10:51:37.750199 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:37.750108    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:40.291411 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:40.291368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" event={"ID":"ad094bcf-53e0-4106-a091-6602d1f66130","Type":"ContainerStarted","Data":"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"}
Jul 29 10:51:40.291775 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:40.291494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:51:40.307781 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:40.307730    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" podStartSLOduration=1.9173103820000001 podStartE2EDuration="4.307712561s" totalImagesPullingTime="2.390402179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:51:40.306350896 +0000 UTC m=+1376.486732777" watchObservedRunningTime="2026-07-29 10:51:40.307712561 +0000 UTC m=+1376.488094441"
Jul 29 10:51:46.750497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:46.750461    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:46.763107 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:46.763079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:51:47.338091 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:51:47.338052    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:52:11.297948 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:11.297919    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:52:12.461814 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.461741    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:52:12.464370 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.464342    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:52:12.474851 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:52:12.474817    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:52:12.474851 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:52:12.474851 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:52:12.474851 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d" cmd=["/bin/sleep","15"]
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.474867    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" message="" err=<
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:52:12.474882    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" containerName="main" containerID="cri-o://e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"
Jul 29 10:52:12.475181 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.474921    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" containerName="main" containerID="cri-o://e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d" gracePeriod=60
Jul 29 10:52:12.824367 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.824344    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:52:12.894341 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894304    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkbn\" (UniqueName: \"kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894367    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894390    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894471    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894564 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894536    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs\") pod \"200087e0-1b7c-40a7-851b-ef45b3352661\" (UID: \"200087e0-1b7c-40a7-851b-ef45b3352661\") "
Jul 29 10:52:12.894844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894655    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache" (OuterVolumeSpecName: "model-cache") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:52:12.894844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894768    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home" (OuterVolumeSpecName: "home") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:52:12.894844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894775    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:52:12.894844 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.894787    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:12.896853 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.896821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm" (OuterVolumeSpecName: "dshm") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:52:12.896988 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.896917    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn" (OuterVolumeSpecName: "kube-api-access-nxkbn") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "kube-api-access-nxkbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:52:12.896988 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.896952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "200087e0-1b7c-40a7-851b-ef45b3352661" (UID: "200087e0-1b7c-40a7-851b-ef45b3352661"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:52:12.996064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.995965    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:12.996064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.996001    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200087e0-1b7c-40a7-851b-ef45b3352661-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:12.996064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.996011    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkbn\" (UniqueName: \"kubernetes.io/projected/200087e0-1b7c-40a7-851b-ef45b3352661-kube-api-access-nxkbn\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:12.996064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.996019    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:12.996064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:12.996028    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/200087e0-1b7c-40a7-851b-ef45b3352661-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:13.455489 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.455452    2574 generic.go:356] "Generic (PLEG): container finished" podID="200087e0-1b7c-40a7-851b-ef45b3352661" containerID="e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d" exitCode=0
Jul 29 10:52:13.455675 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.455510    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"
Jul 29 10:52:13.455675 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.455531    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" event={"ID":"200087e0-1b7c-40a7-851b-ef45b3352661","Type":"ContainerDied","Data":"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"}
Jul 29 10:52:13.455675 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.455583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44" event={"ID":"200087e0-1b7c-40a7-851b-ef45b3352661","Type":"ContainerDied","Data":"c28ef2bf0e97699263b7ffd4ced015f56e606ed14ac181fe4144d8b7f6bab760"}
Jul 29 10:52:13.455675 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.455607    2574 scope.go:117] "RemoveContainer" containerID="e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"
Jul 29 10:52:13.465607 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.465590    2574 scope.go:117] "RemoveContainer" containerID="e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"
Jul 29 10:52:13.465937 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:52:13.465844    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d\": container with ID starting with e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d not found: ID does not exist" containerID="e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"
Jul 29 10:52:13.465937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.465868    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d"} err="failed to get container status \"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d\": rpc error: code = NotFound desc = could not find container \"e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d\": container with ID starting with e3892c24dd9083dcc6afc2fe778780edb01b3469479aa570632981d639cb4d2d not found: ID does not exist"
Jul 29 10:52:13.476536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.476501    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:52:13.484116 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:13.484089    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-hbk44"]
Jul 29 10:52:14.342261 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:14.342211    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" path="/var/lib/kubelet/pods/200087e0-1b7c-40a7-851b-ef45b3352661/volumes"
Jul 29 10:52:21.298152 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:21.298115    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 29 10:52:23.799084 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:23.799051    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 29 10:52:23.799547 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:52:23.799089    2574 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main" containerID="cri-o://d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"
Jul 29 10:52:23.799547 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:23.799139    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main" containerID="cri-o://d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50" gracePeriod=49
Jul 29 10:52:23.960761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:23.960738    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf_ad094bcf-53e0-4106-a091-6602d1f66130/main/0.log"
Jul 29 10:52:23.963011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:23.962995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:52:24.093672 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.093576    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs\") pod \"ad094bcf-53e0-4106-a091-6602d1f66130\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") "
Jul 29 10:52:24.093672 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.093613    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlmtt\" (UniqueName: \"kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt\") pod \"ad094bcf-53e0-4106-a091-6602d1f66130\" (UID: \"ad094bcf-53e0-4106-a091-6602d1f66130\") "
Jul 29 10:52:24.095829 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.095799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad094bcf-53e0-4106-a091-6602d1f66130" (UID: "ad094bcf-53e0-4106-a091-6602d1f66130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:52:24.095829 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.095820    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt" (OuterVolumeSpecName: "kube-api-access-tlmtt") pod "ad094bcf-53e0-4106-a091-6602d1f66130" (UID: "ad094bcf-53e0-4106-a091-6602d1f66130"). InnerVolumeSpecName "kube-api-access-tlmtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:52:24.194730 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.194691    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad094bcf-53e0-4106-a091-6602d1f66130-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:24.194730 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.194722    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlmtt\" (UniqueName: \"kubernetes.io/projected/ad094bcf-53e0-4106-a091-6602d1f66130-kube-api-access-tlmtt\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:52:24.514753 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514721    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf_ad094bcf-53e0-4106-a091-6602d1f66130/main/0.log"
Jul 29 10:52:24.514915 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514765    2574 generic.go:356] "Generic (PLEG): container finished" podID="ad094bcf-53e0-4106-a091-6602d1f66130" containerID="d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50" exitCode=1
Jul 29 10:52:24.514915 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514812    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"
Jul 29 10:52:24.514915 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514858    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" event={"ID":"ad094bcf-53e0-4106-a091-6602d1f66130","Type":"ContainerDied","Data":"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"}
Jul 29 10:52:24.514915 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514908    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf" event={"ID":"ad094bcf-53e0-4106-a091-6602d1f66130","Type":"ContainerDied","Data":"9fae34902796cc6401404d0b4131fb5ff09328dc57a59f01207746d8c94ff941"}
Jul 29 10:52:24.515138 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.514931    2574 scope.go:117] "RemoveContainer" containerID="d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"
Jul 29 10:52:24.523985 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.523970    2574 scope.go:117] "RemoveContainer" containerID="d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"
Jul 29 10:52:24.524286 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:52:24.524264    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50\": container with ID starting with d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50 not found: ID does not exist" containerID="d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"
Jul 29 10:52:24.524336 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.524296    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50"} err="failed to get container status \"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50\": rpc error: code = NotFound desc = could not find container \"d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50\": container with ID starting with d2ee1fc2f7b02a88caf441ebfd185e1961f7bcd457d318706ae7e2ef9b3aba50 not found: ID does not exist"
Jul 29 10:52:24.531330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.531305    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:52:24.535952 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:24.535929    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cffxzmf"]
Jul 29 10:52:26.342553 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:26.342516    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" path="/var/lib/kubelet/pods/ad094bcf-53e0-4106-a091-6602d1f66130/volumes"
Jul 29 10:52:39.536090 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.535990    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536390    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" containerName="main"
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536403    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" containerName="main"
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536417    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main"
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536422    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main"
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536480    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad094bcf-53e0-4106-a091-6602d1f66130" containerName="main"
Jul 29 10:52:39.536495 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.536492    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="200087e0-1b7c-40a7-851b-ef45b3352661" containerName="main"
Jul 29 10:52:39.540741 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.540724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.543508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.543469    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-rg7g2\""
Jul 29 10:52:39.543508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.543493    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 29 10:52:39.543727 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.543516    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 29 10:52:39.543727 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.543521    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 29 10:52:39.547634 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.547603    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:52:39.631880 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.631840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.631880 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.631883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8pq\" (UniqueName: \"kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.632113 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.631922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.632113 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.631966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.632113 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.631995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.632113 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.632014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733039 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8pq\" (UniqueName: \"kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733268 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733619 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733585    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.733677 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.733656    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.735751 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.735727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.735884 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.735734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.741174 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.741149    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8pq\" (UniqueName: \"kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5r6dx\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.852080 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.851990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:39.991954 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:39.991837    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:52:39.994843 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:52:39.994813    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bf5fb0_abe1_4ab8_9bb2_a2727b7f5af4.slice/crio-5df7efeeac9f30d5f5378f418b1115fa32edd2e35e87c08f5ac0966a9d67d5af WatchSource:0}: Error finding container 5df7efeeac9f30d5f5378f418b1115fa32edd2e35e87c08f5ac0966a9d67d5af: Status 404 returned error can't find the container with id 5df7efeeac9f30d5f5378f418b1115fa32edd2e35e87c08f5ac0966a9d67d5af
Jul 29 10:52:40.605542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:40.605500    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" event={"ID":"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4","Type":"ContainerStarted","Data":"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"}
Jul 29 10:52:40.605934 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:40.605565    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" event={"ID":"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4","Type":"ContainerStarted","Data":"5df7efeeac9f30d5f5378f418b1115fa32edd2e35e87c08f5ac0966a9d67d5af"}
Jul 29 10:52:40.622696 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:40.622635    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" podStartSLOduration=1.622613506 podStartE2EDuration="1.622613506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:52:40.622416863 +0000 UTC m=+1436.802798745" watchObservedRunningTime="2026-07-29 10:52:40.622613506 +0000 UTC m=+1436.802995387"
Jul 29 10:52:40.853119 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:40.853084    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:49.853057 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:49.853019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:49.865636 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:49.865611    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:52:50.668105 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:52:50.668063    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:53:12.769189 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:12.769153    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:53:12.781188    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c" cmd=["/bin/sleep","15"]
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:12.781273    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" message="" err=<
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:53:12.781290    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerName="main" containerID="cri-o://7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"
Jul 29 10:53:12.781718 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:12.781346    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerName="main" containerID="cri-o://7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c" gracePeriod=60
Jul 29 10:53:13.135937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.135914    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:53:13.219433 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219395    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219433 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219440    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219488    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219583    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x8pq\" (UniqueName: \"kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219689 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir\") pod \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\" (UID: \"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4\") "
Jul 29 10:53:13.219893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219726    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home" (OuterVolumeSpecName: "home") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:13.219893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.219739    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache" (OuterVolumeSpecName: "model-cache") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:13.220033 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.220002    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:13.220178 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.220159    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.220260 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.220184    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.220316 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.220260    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.221794 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.221774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm" (OuterVolumeSpecName: "dshm") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:53:13.221905 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.221883    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq" (OuterVolumeSpecName: "kube-api-access-7x8pq") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "kube-api-access-7x8pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:53:13.221972 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.221955    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" (UID: "60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:53:13.321458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.321358    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8pq\" (UniqueName: \"kubernetes.io/projected/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-kube-api-access-7x8pq\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.321458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.321395    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.321458 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.321406    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:53:13.776772 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.776741    2574 generic.go:356] "Generic (PLEG): container finished" podID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerID="7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c" exitCode=0
Jul 29 10:53:13.777163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.776788    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"
Jul 29 10:53:13.777163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.776824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" event={"ID":"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4","Type":"ContainerDied","Data":"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"}
Jul 29 10:53:13.777163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.776884    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx" event={"ID":"60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4","Type":"ContainerDied","Data":"5df7efeeac9f30d5f5378f418b1115fa32edd2e35e87c08f5ac0966a9d67d5af"}
Jul 29 10:53:13.777163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.776914    2574 scope.go:117] "RemoveContainer" containerID="7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"
Jul 29 10:53:13.787043 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.786884    2574 scope.go:117] "RemoveContainer" containerID="7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"
Jul 29 10:53:13.787259 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:53:13.787211    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c\": container with ID starting with 7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c not found: ID does not exist" containerID="7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"
Jul 29 10:53:13.787314 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.787271    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c"} err="failed to get container status \"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c\": rpc error: code = NotFound desc = could not find container \"7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c\": container with ID starting with 7dcae1f46c05b727ad720e2fa2e8631e9e11585bc67ba408719f8a89836a4f4c not found: ID does not exist"
Jul 29 10:53:13.800146 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.800102    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:53:13.806633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:13.806603    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5r6dx"]
Jul 29 10:53:14.342628 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:14.342593    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" path="/var/lib/kubelet/pods/60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4/volumes"
Jul 29 10:53:46.052864 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:46.052755    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:53:46.058234 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:53:46.053448    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:54:03.067891 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.067854    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:54:03.068410 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.068386    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerName="main"
Jul 29 10:54:03.068488 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.068413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerName="main"
Jul 29 10:54:03.068550 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.068519    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="60bf5fb0-abe1-4ab8-9bb2-a2727b7f5af4" containerName="main"
Jul 29 10:54:03.071661 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.071639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.075705 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.075673    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 29 10:54:03.076394 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.076295    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 29 10:54:03.076394 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.076305    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 29 10:54:03.076595 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.076578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-87z5c\""
Jul 29 10:54:03.076656 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.076578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-nx2cp\""
Jul 29 10:54:03.091759 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.091730    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:54:03.144530 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.144530 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcwq5\" (UniqueName: \"kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.144807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.144807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144644    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.144807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144678    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.144807 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.144712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.245851 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.245816    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.245851 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.245852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246100 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.245911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246100 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.245927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcwq5\" (UniqueName: \"kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246100 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.245947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246301 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.246097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246364 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.246321    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246422 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.246403    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.246467 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.246440    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.248385 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.248360    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.248550 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.248536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.255818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.255796    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcwq5\" (UniqueName: \"kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.384118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.384017    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:03.531267 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.531034    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:54:03.534067 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:54:03.534035    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2b703c_6eeb_470a_bce8_dec898f33a03.slice/crio-fe27f1e2031bfcedb87cd89c88ea7b7bed5df5f1b13d87009db2d01a47c01d44 WatchSource:0}: Error finding container fe27f1e2031bfcedb87cd89c88ea7b7bed5df5f1b13d87009db2d01a47c01d44: Status 404 returned error can't find the container with id fe27f1e2031bfcedb87cd89c88ea7b7bed5df5f1b13d87009db2d01a47c01d44
Jul 29 10:54:03.536000 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:03.535982    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 10:54:04.034349 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:04.034316    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerStarted","Data":"fe27f1e2031bfcedb87cd89c88ea7b7bed5df5f1b13d87009db2d01a47c01d44"}
Jul 29 10:54:06.046452 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:06.046418    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerStarted","Data":"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"}
Jul 29 10:54:06.046888 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:06.046505    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:07.053857 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:07.053824    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerStarted","Data":"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"}
Jul 29 10:54:07.074205 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:07.074124    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" podStartSLOduration=2.553863336 podStartE2EDuration="4.074106436s" totalImagesPullingTime="1.5202431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:54:07.071295669 +0000 UTC m=+1523.251677549" watchObservedRunningTime="2026-07-29 10:54:07.074106436 +0000 UTC m=+1523.254488316"
Jul 29 10:54:07.384229 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:07.384131    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:13.384906 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:13.384868    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:13.397472 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:13.397443    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:14.098380 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:14.098330    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:54:15.107586 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:54:15.107559    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:55:36.350280 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.350175    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:55:36.361117 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:55:36.361080    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:55:36.361117 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.361117 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.361117 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e" cmd=["/bin/sleep","15"]
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.361134    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" message="" err=<
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:55:36.361147    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="main" containerID="cri-o://2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"
Jul 29 10:55:36.361428 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.361191    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="main" containerID="cri-o://2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e" gracePeriod=60
Jul 29 10:55:36.580637 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.580549    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="llm-d-routing-sidecar" containerID="cri-o://99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4" gracePeriod=60
Jul 29 10:55:36.841935 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.841902    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:55:36.901644 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901611    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.901811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901686    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.901811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901734    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.901811 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901753    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.901980 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.901980 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901866    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcwq5\" (UniqueName: \"kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5\") pod \"3a2b703c-6eeb-470a-bce8-dec898f33a03\" (UID: \"3a2b703c-6eeb-470a-bce8-dec898f33a03\") "
Jul 29 10:55:36.902169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.901987    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.902169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.902082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home" (OuterVolumeSpecName: "home") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.902169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.902132    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:36.902169 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.902134    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.904182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.904151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm" (OuterVolumeSpecName: "dshm") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:55:36.904328 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.904253    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:55:36.904392 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:36.904330    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5" (OuterVolumeSpecName: "kube-api-access-hcwq5") pod "3a2b703c-6eeb-470a-bce8-dec898f33a03" (UID: "3a2b703c-6eeb-470a-bce8-dec898f33a03"). InnerVolumeSpecName "kube-api-access-hcwq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:55:37.002956 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.002855    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.002956 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.002890    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcwq5\" (UniqueName: \"kubernetes.io/projected/3a2b703c-6eeb-470a-bce8-dec898f33a03-kube-api-access-hcwq5\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.002956 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.002900    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2b703c-6eeb-470a-bce8-dec898f33a03-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.002956 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.002909    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.002956 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.002916    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2b703c-6eeb-470a-bce8-dec898f33a03-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:55:37.517610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517573    2574 generic.go:356] "Generic (PLEG): container finished" podID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e" exitCode=0
Jul 29 10:55:37.517610 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517602    2574 generic.go:356] "Generic (PLEG): container finished" podID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerID="99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4" exitCode=0
Jul 29 10:55:37.518193 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517643    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"
Jul 29 10:55:37.518193 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517655    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerDied","Data":"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"}
Jul 29 10:55:37.518193 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517719    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerDied","Data":"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"}
Jul 29 10:55:37.518193 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517740    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm" event={"ID":"3a2b703c-6eeb-470a-bce8-dec898f33a03","Type":"ContainerDied","Data":"fe27f1e2031bfcedb87cd89c88ea7b7bed5df5f1b13d87009db2d01a47c01d44"}
Jul 29 10:55:37.518193 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.517762    2574 scope.go:117] "RemoveContainer" containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"
Jul 29 10:55:37.529150 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.529128    2574 scope.go:117] "RemoveContainer" containerID="99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"
Jul 29 10:55:37.539051 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539030    2574 scope.go:117] "RemoveContainer" containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"
Jul 29 10:55:37.539461 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:55:37.539428    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e\": container with ID starting with 2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e not found: ID does not exist" containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"
Jul 29 10:55:37.539535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539474    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"} err="failed to get container status \"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e\": rpc error: code = NotFound desc = could not find container \"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e\": container with ID starting with 2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e not found: ID does not exist"
Jul 29 10:55:37.539535 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539502    2574 scope.go:117] "RemoveContainer" containerID="99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"
Jul 29 10:55:37.539631 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539611    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:55:37.539842 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:55:37.539823    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4\": container with ID starting with 99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4 not found: ID does not exist" containerID="99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"
Jul 29 10:55:37.539901 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539851    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"} err="failed to get container status \"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4\": rpc error: code = NotFound desc = could not find container \"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4\": container with ID starting with 99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4 not found: ID does not exist"
Jul 29 10:55:37.539901 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.539875    2574 scope.go:117] "RemoveContainer" containerID="2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"
Jul 29 10:55:37.540151 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.540129    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e"} err="failed to get container status \"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e\": rpc error: code = NotFound desc = could not find container \"2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e\": container with ID starting with 2620bdfe74795e9b71adeef675cdee743239dea83d2f8767a86ffc288704ae9e not found: ID does not exist"
Jul 29 10:55:37.540260 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.540155    2574 scope.go:117] "RemoveContainer" containerID="99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"
Jul 29 10:55:37.540503 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.540474    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4"} err="failed to get container status \"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4\": rpc error: code = NotFound desc = could not find container \"99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4\": container with ID starting with 99c431fda955544a16da41c05dd6a87a65ecbee1bb97fcf05f6de8b8a39354e4 not found: ID does not exist"
Jul 29 10:55:37.543129 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:37.543101    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-lp9zm"]
Jul 29 10:55:38.348007 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:55:38.347971    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" path="/var/lib/kubelet/pods/3a2b703c-6eeb-470a-bce8-dec898f33a03/volumes"
Jul 29 10:56:07.977256 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977205    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:07.977855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977707    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="llm-d-routing-sidecar"
Jul 29 10:56:07.977855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977726    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="llm-d-routing-sidecar"
Jul 29 10:56:07.977855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977748    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="main"
Jul 29 10:56:07.977855 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977757    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="main"
Jul 29 10:56:07.978075 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977857    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="main"
Jul 29 10:56:07.978075 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.977870    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2b703c-6eeb-470a-bce8-dec898f33a03" containerName="llm-d-routing-sidecar"
Jul 29 10:56:07.982693 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.982661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:07.989526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.989155    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 29 10:56:07.989526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.989189    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-2kl8q\""
Jul 29 10:56:07.989526 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.989249    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-q52ss\""
Jul 29 10:56:07.989737 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.989624    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 29 10:56:07.989737 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:07.989624    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 29 10:56:08.003182 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.003153    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:08.171107 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.171331 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsh6f\" (UniqueName: \"kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.171331 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.171331 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.171331 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.171331 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.171272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272597 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272685    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.272818 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.272713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsh6f\" (UniqueName: \"kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.273139 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.273071    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.273139 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.273102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.273139 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.273113    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.275199 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.275170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.275501 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.275481    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.283497 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.283470    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsh6f\" (UniqueName: \"kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.292378 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.292353    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.438949 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.438925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:08.441341 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:56:08.441312    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fc3a87_668c_4943_96aa_6b668f7cba1f.slice/crio-116be22d27b934e01b89d7fcabbe1f8f9abe445a46114641aa55db907d142d75 WatchSource:0}: Error finding container 116be22d27b934e01b89d7fcabbe1f8f9abe445a46114641aa55db907d142d75: Status 404 returned error can't find the container with id 116be22d27b934e01b89d7fcabbe1f8f9abe445a46114641aa55db907d142d75
Jul 29 10:56:08.678720 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.678631    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerStarted","Data":"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"}
Jul 29 10:56:08.678720 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.678696    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:08.678720 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:08.678721    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerStarted","Data":"116be22d27b934e01b89d7fcabbe1f8f9abe445a46114641aa55db907d142d75"}
Jul 29 10:56:09.686138 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:09.686099    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerStarted","Data":"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"}
Jul 29 10:56:09.706381 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:09.706324    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" podStartSLOduration=2.7063051099999997 podStartE2EDuration="2.70630511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:56:09.703667955 +0000 UTC m=+1645.884049834" watchObservedRunningTime="2026-07-29 10:56:09.70630511 +0000 UTC m=+1645.886686994"
Jul 29 10:56:10.293200 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:10.293148    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:18.293098 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:18.293057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:18.305752 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:18.305718    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:18.317064 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:18.317031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:18.740671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:18.740585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:42.006354 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.006313    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:42.017136 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:56:42.017092    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:56:42.017136 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.017136 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.017136 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193" cmd=["/bin/sleep","15"]
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.017153    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" message="" err=<
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:56:42.017165    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="main" containerID="cri-o://7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"
Jul 29 10:56:42.017449 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.017239    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="main" containerID="cri-o://7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193" gracePeriod=60
Jul 29 10:56:42.132549 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.132497    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="llm-d-routing-sidecar" containerID="cri-o://5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7" gracePeriod=60
Jul 29 10:56:42.385074 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.385044    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:42.457599 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457564    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.457599 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457601    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.457852 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457620    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.457852 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457683    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsh6f\" (UniqueName: \"kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.457852 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457706    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.457852 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457751    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home\") pod \"05fc3a87-668c-4943-96aa-6b668f7cba1f\" (UID: \"05fc3a87-668c-4943-96aa-6b668f7cba1f\") "
Jul 29 10:56:42.458056 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457872    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.458056 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.457953    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.458056 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.458000    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.458177 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.458085    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home" (OuterVolumeSpecName: "home") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.460096 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.460066    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:56:42.460211 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.460094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm" (OuterVolumeSpecName: "dshm") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:56:42.460211 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.460107    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f" (OuterVolumeSpecName: "kube-api-access-lsh6f") pod "05fc3a87-668c-4943-96aa-6b668f7cba1f" (UID: "05fc3a87-668c-4943-96aa-6b668f7cba1f"). InnerVolumeSpecName "kube-api-access-lsh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:56:42.558847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.558759    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.558847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.558789    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fc3a87-668c-4943-96aa-6b668f7cba1f-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.558847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.558804    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.558847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.558815    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsh6f\" (UniqueName: \"kubernetes.io/projected/05fc3a87-668c-4943-96aa-6b668f7cba1f-kube-api-access-lsh6f\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.558847 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.558826    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fc3a87-668c-4943-96aa-6b668f7cba1f-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:56:42.855542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855448    2574 generic.go:356] "Generic (PLEG): container finished" podID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193" exitCode=0
Jul 29 10:56:42.855542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855473    2574 generic.go:356] "Generic (PLEG): container finished" podID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerID="5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7" exitCode=0
Jul 29 10:56:42.855542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855501    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"
Jul 29 10:56:42.855542 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerDied","Data":"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"}
Jul 29 10:56:42.855812 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855597    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerDied","Data":"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"}
Jul 29 10:56:42.855812 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855624    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6" event={"ID":"05fc3a87-668c-4943-96aa-6b668f7cba1f","Type":"ContainerDied","Data":"116be22d27b934e01b89d7fcabbe1f8f9abe445a46114641aa55db907d142d75"}
Jul 29 10:56:42.855812 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.855654    2574 scope.go:117] "RemoveContainer" containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"
Jul 29 10:56:42.867671 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.867645    2574 scope.go:117] "RemoveContainer" containerID="5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"
Jul 29 10:56:42.878109 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878085    2574 scope.go:117] "RemoveContainer" containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"
Jul 29 10:56:42.878524 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:56:42.878491    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193\": container with ID starting with 7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193 not found: ID does not exist" containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"
Jul 29 10:56:42.878651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878532    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"} err="failed to get container status \"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193\": rpc error: code = NotFound desc = could not find container \"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193\": container with ID starting with 7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193 not found: ID does not exist"
Jul 29 10:56:42.878651 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878560    2574 scope.go:117] "RemoveContainer" containerID="5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"
Jul 29 10:56:42.878875 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:56:42.878854    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7\": container with ID starting with 5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7 not found: ID does not exist" containerID="5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"
Jul 29 10:56:42.878937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878881    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"} err="failed to get container status \"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7\": rpc error: code = NotFound desc = could not find container \"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7\": container with ID starting with 5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7 not found: ID does not exist"
Jul 29 10:56:42.878937 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878902    2574 scope.go:117] "RemoveContainer" containerID="7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"
Jul 29 10:56:42.879029 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.878945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:42.879157 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.879128    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193"} err="failed to get container status \"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193\": rpc error: code = NotFound desc = could not find container \"7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193\": container with ID starting with 7f1f5bd30c9a55f5109ada71e2e7582adb8bd2c505bf9bfde6c60c1e84770193 not found: ID does not exist"
Jul 29 10:56:42.879203 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.879162    2574 scope.go:117] "RemoveContainer" containerID="5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"
Jul 29 10:56:42.879509 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.879467    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7"} err="failed to get container status \"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7\": rpc error: code = NotFound desc = could not find container \"5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7\": container with ID starting with 5af886d1b10accb41529192fb9a4943a92f77b50ab0ec74b7e75b35424d5f5b7 not found: ID does not exist"
Jul 29 10:56:42.884355 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:42.884324    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-6dpb6"]
Jul 29 10:56:44.341683 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:56:44.341648    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" path="/var/lib/kubelet/pods/05fc3a87-668c-4943-96aa-6b668f7cba1f/volumes"
Jul 29 10:57:08.062560 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062478    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062805    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="llm-d-routing-sidecar"
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062816    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="llm-d-routing-sidecar"
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062828    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="main"
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062834    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="main"
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062892    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="llm-d-routing-sidecar"
Jul 29 10:57:08.062930 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.062899    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="05fc3a87-668c-4943-96aa-6b668f7cba1f" containerName="main"
Jul 29 10:57:08.067106 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.067083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.069604 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.069566    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 29 10:57:08.069761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.069610    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 29 10:57:08.069761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.069630    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 29 10:57:08.069761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.069610    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 29 10:57:08.069761 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.069687    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-gj6v8\""
Jul 29 10:57:08.077465 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.077437    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:57:08.078886 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.078861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.078926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.078953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079005 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.078990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmhtg\" (UniqueName: \"kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.079025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.079057    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.079143 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.079129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180359 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmhtg\" (UniqueName: \"kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180786    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180876 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180896    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.180989 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.180973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.183048 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.183024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.183296 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.183279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.187955 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.187931    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmhtg\" (UniqueName: \"kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg\") pod \"tokenizer-clean-path-test-kserve-768597f86d-fmmmb\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.378467 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.378373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:08.530129 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.530094    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:57:08.533777 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:57:08.533748    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941e912b_1737_4db5_aaea_b9d3a61c8dd3.slice/crio-de3018af1f294d0994d9b2ac879c80f49f583c8bee5b76a41a63d4f23c7ae83d WatchSource:0}: Error finding container de3018af1f294d0994d9b2ac879c80f49f583c8bee5b76a41a63d4f23c7ae83d: Status 404 returned error can't find the container with id de3018af1f294d0994d9b2ac879c80f49f583c8bee5b76a41a63d4f23c7ae83d
Jul 29 10:57:08.989003 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.988967    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerStarted","Data":"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"}
Jul 29 10:57:08.989183 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:08.989022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerStarted","Data":"de3018af1f294d0994d9b2ac879c80f49f583c8bee5b76a41a63d4f23c7ae83d"}
Jul 29 10:57:13.022609 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:13.022572    2574 generic.go:356] "Generic (PLEG): container finished" podID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerID="3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8" exitCode=0
Jul 29 10:57:13.023153 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:13.022628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerDied","Data":"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"}
Jul 29 10:57:14.030013 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:14.029973    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerStarted","Data":"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"}
Jul 29 10:57:14.049529 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:14.049455    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" podStartSLOduration=1.7989965140000002 podStartE2EDuration="6.049436221s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.250439707s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:57:14.047314943 +0000 UTC m=+1710.227696824" watchObservedRunningTime="2026-07-29 10:57:14.049436221 +0000 UTC m=+1710.229818101"
Jul 29 10:57:14.379529 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:14.379439    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:18.379426 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:18.379383    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:18.392352 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:18.392323    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:57:19.065756 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:57:19.065719    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:58:19.479666 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.479624    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:58:19.492974 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:58:19.492897    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:58:19.492974 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.492974 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.492974 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377" cmd=["/bin/sleep","15"]
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.492989    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" message="" err=<
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:58:19.493010    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="main" containerID="cri-o://8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"
Jul 29 10:58:19.493330 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.493087    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="main" containerID="cri-o://8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377" gracePeriod=60
Jul 29 10:58:19.859469 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.859439    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:58:19.925418 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925374    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925472    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925500    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmhtg\" (UniqueName: \"kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925572    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925633 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925609    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925870 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925667    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm\") pod \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\" (UID: \"941e912b-1737-4db5-aaea-b9d3a61c8dd3\") "
Jul 29 10:58:19.925870 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.925870 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925762    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home" (OuterVolumeSpecName: "home") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.926009 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925926    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:19.926009 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.925947    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:19.926120 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.926019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.928163 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.928136    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm" (OuterVolumeSpecName: "dshm") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:19.928306 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.928261    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg" (OuterVolumeSpecName: "kube-api-access-mmhtg") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "kube-api-access-mmhtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:58:19.928368 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.928351    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:58:19.985504 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:19.985454    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "941e912b-1737-4db5-aaea-b9d3a61c8dd3" (UID: "941e912b-1737-4db5-aaea-b9d3a61c8dd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:58:20.027128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.027085    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.027128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.027122    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmhtg\" (UniqueName: \"kubernetes.io/projected/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kube-api-access-mmhtg\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.027363 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.027138    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/941e912b-1737-4db5-aaea-b9d3a61c8dd3-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.027363 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.027154    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.027363 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.027166    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/941e912b-1737-4db5-aaea-b9d3a61c8dd3-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:58:20.371891 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.371853    2574 generic.go:356] "Generic (PLEG): container finished" podID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerID="8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377" exitCode=0
Jul 29 10:58:20.372069 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.371922    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerDied","Data":"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"}
Jul 29 10:58:20.372069 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.371953    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"
Jul 29 10:58:20.372069 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.371970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb" event={"ID":"941e912b-1737-4db5-aaea-b9d3a61c8dd3","Type":"ContainerDied","Data":"de3018af1f294d0994d9b2ac879c80f49f583c8bee5b76a41a63d4f23c7ae83d"}
Jul 29 10:58:20.372069 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.371999    2574 scope.go:117] "RemoveContainer" containerID="8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"
Jul 29 10:58:20.381383 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.381362    2574 scope.go:117] "RemoveContainer" containerID="3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"
Jul 29 10:58:20.389973 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.389951    2574 scope.go:117] "RemoveContainer" containerID="8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"
Jul 29 10:58:20.390296 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:58:20.390266    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377\": container with ID starting with 8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377 not found: ID does not exist" containerID="8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"
Jul 29 10:58:20.390467 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.390302    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377"} err="failed to get container status \"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377\": rpc error: code = NotFound desc = could not find container \"8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377\": container with ID starting with 8a41f28ba221496ebdd24437be88ebbad02cd40993a66a43e8e2b36b7a45c377 not found: ID does not exist"
Jul 29 10:58:20.390467 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.390327    2574 scope.go:117] "RemoveContainer" containerID="3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"
Jul 29 10:58:20.390935 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:58:20.390672    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8\": container with ID starting with 3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8 not found: ID does not exist" containerID="3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"
Jul 29 10:58:20.390935 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.390703    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8"} err="failed to get container status \"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8\": rpc error: code = NotFound desc = could not find container \"3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8\": container with ID starting with 3337e7ab052c36d9af0707395f75dc467138adaa4876e7f68156dd6fc8d28bb8 not found: ID does not exist"
Jul 29 10:58:20.393385 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.393327    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:58:20.395286 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:20.395268    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-768597f86d-fmmmb"]
Jul 29 10:58:22.341882 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:22.341849    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" path="/var/lib/kubelet/pods/941e912b-1737-4db5-aaea-b9d3a61c8dd3/volumes"
Jul 29 10:58:46.075101 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.074939    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.075420    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="main"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.075440    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="main"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.075465    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="storage-initializer"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.075475    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="storage-initializer"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.075564    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="941e912b-1737-4db5-aaea-b9d3a61c8dd3" containerName="main"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.078924    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.082543    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-wftkd\""
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.082630    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.082792    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.083726    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.083845    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.091570    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:58:46.095642 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.095190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:58:46.097069 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.095684    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 10:58:46.155825 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.155786    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.155853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.155879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.155913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156011 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.155972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.156011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.156156 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.156029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hnm\" (UniqueName: \"kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256484 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hnm\" (UniqueName: \"kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256708 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.256997 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.257053 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.256995    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.257053 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.257024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.257128 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.257055    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.259345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.259318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.259494 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.259392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.265830 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.265802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hnm\" (UniqueName: \"kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm\") pod \"tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.393503 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.393397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:46.542765 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:46.542741    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:58:46.545015 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:58:46.544986    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b57491_b458_4cf8_b348_51c9fb6b7502.slice/crio-19d523e298d7bba20caa9bc0456c59b4a62fd7057ee952ed25da64137affb65d WatchSource:0}: Error finding container 19d523e298d7bba20caa9bc0456c59b4a62fd7057ee952ed25da64137affb65d: Status 404 returned error can't find the container with id 19d523e298d7bba20caa9bc0456c59b4a62fd7057ee952ed25da64137affb65d
Jul 29 10:58:47.516431 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:47.516386    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerStarted","Data":"22050cbe475f1f233c49ba8bc6c1da2633d3e17987b0b98aa935cadba72312d2"}
Jul 29 10:58:47.516856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:47.516492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerStarted","Data":"19d523e298d7bba20caa9bc0456c59b4a62fd7057ee952ed25da64137affb65d"}
Jul 29 10:58:51.540442 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:51.540403    2574 generic.go:356] "Generic (PLEG): container finished" podID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerID="22050cbe475f1f233c49ba8bc6c1da2633d3e17987b0b98aa935cadba72312d2" exitCode=0
Jul 29 10:58:51.540891 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:51.540480    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerDied","Data":"22050cbe475f1f233c49ba8bc6c1da2633d3e17987b0b98aa935cadba72312d2"}
Jul 29 10:58:52.548072 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:52.548031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerStarted","Data":"f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445"}
Jul 29 10:58:52.566329 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:52.566275    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" podStartSLOduration=2.313058258 podStartE2EDuration="6.566256377s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.253198119s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:58:52.564520131 +0000 UTC m=+1808.744902050" watchObservedRunningTime="2026-07-29 10:58:52.566256377 +0000 UTC m=+1808.746638260"
Jul 29 10:58:53.393804 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:53.393761    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:56.393849 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:56.393805    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:56.406831 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:56.406793    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:58:56.580523 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:58:56.580489    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:59:24.587757 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.587717    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:59:24.599452 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:59:24.599410    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 10:59:24.599452 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.599452 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.599452 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445" cmd=["/bin/sleep","15"]
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.599467    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" message="" err=<
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]: E0729 10:59:24.599480    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="main" containerID="cri-o://f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445"
Jul 29 10:59:24.599849 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.599517    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="main" containerID="cri-o://f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445" gracePeriod=60
Jul 29 10:59:24.722877 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.722846    2574 generic.go:356] "Generic (PLEG): container finished" podID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerID="f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445" exitCode=0
Jul 29 10:59:24.723037 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.722900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerDied","Data":"f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445"}
Jul 29 10:59:24.849246 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.849162    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:59:24.990473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990437    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990473 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990500    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990527    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990564    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66hnm\" (UniqueName: \"kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.990742 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location\") pod \"58b57491-b458-4cf8-b348-51c9fb6b7502\" (UID: \"58b57491-b458-4cf8-b348-51c9fb6b7502\") "
Jul 29 10:59:24.991012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990815    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home" (OuterVolumeSpecName: "home") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:24.991012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990892    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache" (OuterVolumeSpecName: "model-cache") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:24.991012 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.990933    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:24.993007 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.992977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm" (OuterVolumeSpecName: "dshm") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:24.993118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.993034    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 10:59:24.993118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:24.993057    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm" (OuterVolumeSpecName: "kube-api-access-66hnm") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "kube-api-access-66hnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 10:59:25.056073 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.056031    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58b57491-b458-4cf8-b348-51c9fb6b7502" (UID: "58b57491-b458-4cf8-b348-51c9fb6b7502"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 10:59:25.092114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092077    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092106    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092114 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092116    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hnm\" (UniqueName: \"kubernetes.io/projected/58b57491-b458-4cf8-b348-51c9fb6b7502-kube-api-access-66hnm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092128    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092137    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092145    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58b57491-b458-4cf8-b348-51c9fb6b7502-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.092384 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.092153    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58b57491-b458-4cf8-b348-51c9fb6b7502-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 10:59:25.729874 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.729831    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx" event={"ID":"58b57491-b458-4cf8-b348-51c9fb6b7502","Type":"ContainerDied","Data":"19d523e298d7bba20caa9bc0456c59b4a62fd7057ee952ed25da64137affb65d"}
Jul 29 10:59:25.730317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.729896    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"
Jul 29 10:59:25.730317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.729912    2574 scope.go:117] "RemoveContainer" containerID="f517f496e27fd3e7d7847f28c5c76695d26429c93d18d514d120d6c0e2432445"
Jul 29 10:59:25.739685 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.739668    2574 scope.go:117] "RemoveContainer" containerID="22050cbe475f1f233c49ba8bc6c1da2633d3e17987b0b98aa935cadba72312d2"
Jul 29 10:59:25.750181 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.750154    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:59:25.757276 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:25.757243    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5dd8cffcc-sxzhx"]
Jul 29 10:59:26.343851 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:26.343812    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" path="/var/lib/kubelet/pods/58b57491-b458-4cf8-b348-51c9fb6b7502/volumes"
Jul 29 10:59:48.827481 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827447    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 10:59:48.827893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827781    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="storage-initializer"
Jul 29 10:59:48.827893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827795    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="storage-initializer"
Jul 29 10:59:48.827893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827811    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="main"
Jul 29 10:59:48.827893 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827820    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="main"
Jul 29 10:59:48.828035 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.827897    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58b57491-b458-4cf8-b348-51c9fb6b7502" containerName="main"
Jul 29 10:59:48.830385 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.830366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.832790 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.832763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 29 10:59:48.832945 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.832800    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 29 10:59:48.832945 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.832874    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-626tr\""
Jul 29 10:59:48.832945 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.832912    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 29 10:59:48.839536 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.839505    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 10:59:48.899345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.899345 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.899578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.899578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899398    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.899578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:48.899578 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:48.899467    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p98b\" (UniqueName: \"kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000653 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.000856 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.000796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p98b\" (UniqueName: \"kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.001118 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.001094    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.001177 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.001133    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.001177 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.001158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.003508 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.003479    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.003804 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.003785    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.008858 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.008829    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p98b\" (UniqueName: \"kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.141317 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.141199    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:49.287966 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.287939    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 10:59:49.290098 ip-10-0-133-86 kubenswrapper[2574]: W0729 10:59:49.290071    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2e113c_f705_4737_b484_74044b5cc303.slice/crio-c0ad98660572f4cf9604a6789963d2f5759e6fdbcb9644e49ef3793994d6d218 WatchSource:0}: Error finding container c0ad98660572f4cf9604a6789963d2f5759e6fdbcb9644e49ef3793994d6d218: Status 404 returned error can't find the container with id c0ad98660572f4cf9604a6789963d2f5759e6fdbcb9644e49ef3793994d6d218
Jul 29 10:59:49.854713 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.854676    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" event={"ID":"9c2e113c-f705-4737-b484-74044b5cc303","Type":"ContainerStarted","Data":"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"}
Jul 29 10:59:49.855264 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.854726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" event={"ID":"9c2e113c-f705-4737-b484-74044b5cc303","Type":"ContainerStarted","Data":"c0ad98660572f4cf9604a6789963d2f5759e6fdbcb9644e49ef3793994d6d218"}
Jul 29 10:59:49.871008 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:49.870943    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" podStartSLOduration=1.8709253829999999 podStartE2EDuration="1.870925383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 10:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 10:59:49.868715655 +0000 UTC m=+1866.049097536" watchObservedRunningTime="2026-07-29 10:59:49.870925383 +0000 UTC m=+1866.051307262"
Jul 29 10:59:50.141827 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:50.141747    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:59.141996 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:59.141959    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:59.154537 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:59.154509    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 10:59:59.915312 ip-10-0-133-86 kubenswrapper[2574]: I0729 10:59:59.915274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 11:00:22.072030 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.071952    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 11:00:22.081389 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:00:22.081355    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:00:22.081389 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:00:22.081389 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:00:22.081389 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395" cmd=["/bin/sleep","15"]
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.081403    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" message="" err=<
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:00:22.081415    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" podUID="9c2e113c-f705-4737-b484-74044b5cc303" containerName="main" containerID="cri-o://06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"
Jul 29 11:00:22.081721 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.081458    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" podUID="9c2e113c-f705-4737-b484-74044b5cc303" containerName="main" containerID="cri-o://06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395" gracePeriod=60
Jul 29 11:00:22.440408 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.440377    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594711    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594778    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9p98b\" (UniqueName: \"kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594903    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594926    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.595490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.594949    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs\") pod \"9c2e113c-f705-4737-b484-74044b5cc303\" (UID: \"9c2e113c-f705-4737-b484-74044b5cc303\") "
Jul 29 11:00:22.596420 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.596392    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home" (OuterVolumeSpecName: "home") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:00:22.596683 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.596658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:00:22.596835 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.596701    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:00:22.598456 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.598419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b" (OuterVolumeSpecName: "kube-api-access-9p98b") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "kube-api-access-9p98b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:00:22.601394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.601340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:00:22.602176 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.602153    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm" (OuterVolumeSpecName: "dshm") pod "9c2e113c-f705-4737-b484-74044b5cc303" (UID: "9c2e113c-f705-4737-b484-74044b5cc303"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:00:22.696291 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696260    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:22.696291 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696290    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:22.696291 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696299    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p98b\" (UniqueName: \"kubernetes.io/projected/9c2e113c-f705-4737-b484-74044b5cc303-kube-api-access-9p98b\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:22.696523 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696309    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:22.696523 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696318    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2e113c-f705-4737-b484-74044b5cc303-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:22.696523 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:22.696326    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2e113c-f705-4737-b484-74044b5cc303-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:00:23.023705 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.023676    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c2e113c-f705-4737-b484-74044b5cc303" containerID="06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395" exitCode=0
Jul 29 11:00:23.023876 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.023724    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"
Jul 29 11:00:23.023876 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.023766    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" event={"ID":"9c2e113c-f705-4737-b484-74044b5cc303","Type":"ContainerDied","Data":"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"}
Jul 29 11:00:23.023876 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.023823    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m" event={"ID":"9c2e113c-f705-4737-b484-74044b5cc303","Type":"ContainerDied","Data":"c0ad98660572f4cf9604a6789963d2f5759e6fdbcb9644e49ef3793994d6d218"}
Jul 29 11:00:23.023876 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.023846    2574 scope.go:117] "RemoveContainer" containerID="06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"
Jul 29 11:00:23.033185 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.033168    2574 scope.go:117] "RemoveContainer" containerID="06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"
Jul 29 11:00:23.033469 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:00:23.033451    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395\": container with ID starting with 06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395 not found: ID does not exist" containerID="06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"
Jul 29 11:00:23.033540 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.033486    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395"} err="failed to get container status \"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395\": rpc error: code = NotFound desc = could not find container \"06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395\": container with ID starting with 06dbe7e16c41e0366347012bb52464b62a9011f7857a6d7a19907cccf2990395 not found: ID does not exist"
Jul 29 11:00:23.044429 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.044403    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 11:00:23.048253 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:23.048235    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-vbn6m"]
Jul 29 11:00:24.341837 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:24.341805    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2e113c-f705-4737-b484-74044b5cc303" path="/var/lib/kubelet/pods/9c2e113c-f705-4737-b484-74044b5cc303/volumes"
Jul 29 11:00:58.028844 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.028803    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:00:58.029365 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.029348    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2e113c-f705-4737-b484-74044b5cc303" containerName="main"
Jul 29 11:00:58.029414 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.029368    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2e113c-f705-4737-b484-74044b5cc303" containerName="main"
Jul 29 11:00:58.029490 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.029478    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2e113c-f705-4737-b484-74044b5cc303" containerName="main"
Jul 29 11:00:58.032868 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.032848    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.038508 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.038087    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 29 11:00:58.038508 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.038281    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9rr5h\""
Jul 29 11:00:58.038508 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.038387    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 29 11:00:58.038508 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.038507    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 29 11:00:58.041233 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.041192    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:00:58.093952 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.093915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.094135 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.093969    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.094135 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.094026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.094135 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.094065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.094135 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.094083    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vll44\" (UniqueName: \"kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.094135 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.094103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195156 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195169    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vll44\" (UniqueName: \"kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195394 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195664 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195616    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195664 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.195827 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.195808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.197823 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.197799    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.197979 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.197962    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.202591 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.202566    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vll44\" (UniqueName: \"kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.344782 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.344685    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:00:58.494235 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:58.494187    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:00:58.497420 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:00:58.497389    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bb451d_6f4c_4e16_a691_d58d35c0eeb8.slice/crio-5d8818a2256e2526047ea66377030ed1a8a49ea89156813013ffb467e7bb6f87 WatchSource:0}: Error finding container 5d8818a2256e2526047ea66377030ed1a8a49ea89156813013ffb467e7bb6f87: Status 404 returned error can't find the container with id 5d8818a2256e2526047ea66377030ed1a8a49ea89156813013ffb467e7bb6f87
Jul 29 11:00:59.206572 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:59.206532    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" event={"ID":"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8","Type":"ContainerStarted","Data":"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"}
Jul 29 11:00:59.207051 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:59.206601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" event={"ID":"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8","Type":"ContainerStarted","Data":"5d8818a2256e2526047ea66377030ed1a8a49ea89156813013ffb467e7bb6f87"}
Jul 29 11:00:59.222165 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:59.222100    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" podStartSLOduration=1.222078504 podStartE2EDuration="1.222078504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:00:59.221310559 +0000 UTC m=+1935.401692440" watchObservedRunningTime="2026-07-29 11:00:59.222078504 +0000 UTC m=+1935.402460398"
Jul 29 11:00:59.344848 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:00:59.344813    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:01:08.345275 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:01:08.345236    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:01:08.358102 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:01:08.358081    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:01:09.274420 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:01:09.274388    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:02:28.040602 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.040567    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:02:28.051267 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:02:28.051230    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:02:28.051267 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:28.051267 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:28.051267 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be" cmd=["/bin/sleep","15"]
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.051290    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" message="" err=<
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:02:28.051304    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main" containerID="cri-o://cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"
Jul 29 11:02:28.051542 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.051359    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main" containerID="cri-o://cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be" gracePeriod=60
Jul 29 11:02:28.263275 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.263198    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
Jul 29 11:02:28.417597 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.417575    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:02:28.570653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570618    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.570836 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.570836 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570696    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vll44\" (UniqueName: \"kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.570836 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570732    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.570836 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570777    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.571061 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570891    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home" (OuterVolumeSpecName: "home") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:28.571061 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.570931    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs\") pod \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\" (UID: \"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8\") "
Jul 29 11:02:28.571061 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.571017    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:28.571196 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.571118    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:28.571270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.571259    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:28.571336 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.571279    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:28.571336 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.571294    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:28.573024 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.572960    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm" (OuterVolumeSpecName: "dshm") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:02:28.573131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.573033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44" (OuterVolumeSpecName: "kube-api-access-vll44") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "kube-api-access-vll44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:02:28.573131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.573059    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" (UID: "c0bb451d-6f4c-4e16-a691-d58d35c0eeb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:02:28.644983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.644946    2574 generic.go:356] "Generic (PLEG): container finished" podID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerID="cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be" exitCode=0
Jul 29 11:02:28.645133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.644994    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"
Jul 29 11:02:28.645133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.645023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" event={"ID":"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8","Type":"ContainerDied","Data":"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"}
Jul 29 11:02:28.645133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.645072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt" event={"ID":"c0bb451d-6f4c-4e16-a691-d58d35c0eeb8","Type":"ContainerDied","Data":"5d8818a2256e2526047ea66377030ed1a8a49ea89156813013ffb467e7bb6f87"}
Jul 29 11:02:28.645133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.645094    2574 scope.go:117] "RemoveContainer" containerID="cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"
Jul 29 11:02:28.654477 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.654457    2574 scope.go:117] "RemoveContainer" containerID="cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"
Jul 29 11:02:28.654751 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:02:28.654733    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be\": container with ID starting with cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be not found: ID does not exist" containerID="cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"
Jul 29 11:02:28.654809 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.654759    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be"} err="failed to get container status \"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be\": rpc error: code = NotFound desc = could not find container \"cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be\": container with ID starting with cfc19f88790a7f0170f11bd16fbc1a45f8f2cde097ae01726d6dd6b0eebd71be not found: ID does not exist"
Jul 29 11:02:28.665339 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.665311    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:02:28.670614 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.670594    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5qvqt"]
Jul 29 11:02:28.671824 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.671807    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:28.671885 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.671826    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:28.671885 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:28.671835    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vll44\" (UniqueName: \"kubernetes.io/projected/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8-kube-api-access-vll44\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:02:30.342152 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:30.342123    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" path="/var/lib/kubelet/pods/c0bb451d-6f4c-4e16-a691-d58d35c0eeb8/volumes"
Jul 29 11:02:53.027986 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.027944    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:02:53.028419 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.028354    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main"
Jul 29 11:02:53.028419 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.028372    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main"
Jul 29 11:02:53.028500 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.028452    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0bb451d-6f4c-4e16-a691-d58d35c0eeb8" containerName="main"
Jul 29 11:02:53.031242 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.031209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.033529 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.033507    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 29 11:02:53.034643 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.034625    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 29 11:02:53.034726 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.034646    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 29 11:02:53.034782 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.034626    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-2rqx9\""
Jul 29 11:02:53.038953 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.038931    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:02:53.135923 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.135897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.136034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.135931    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96ml8\" (UniqueName: \"kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.136034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.135965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.136109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.136039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.136109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.136070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.136109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.136095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237115 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237233 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237233 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237213    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237448 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96ml8\" (UniqueName: \"kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237583 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237636 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.237875 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.237854    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.240270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.240248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.240379 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.240362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.246021 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.246001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96ml8\" (UniqueName: \"kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.343144 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.343093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:02:53.469293 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.469162    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:02:53.471580 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:02:53.471553    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eb6eb8_3fab_4327_9545_392ea1d79418.slice/crio-68fdbe388c68c448b7e61fdb86af6a0b9aa61b133e857e9fde9518a0041cd22a WatchSource:0}: Error finding container 68fdbe388c68c448b7e61fdb86af6a0b9aa61b133e857e9fde9518a0041cd22a: Status 404 returned error can't find the container with id 68fdbe388c68c448b7e61fdb86af6a0b9aa61b133e857e9fde9518a0041cd22a
Jul 29 11:02:53.762540 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.762454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" event={"ID":"b5eb6eb8-3fab-4327-9545-392ea1d79418","Type":"ContainerStarted","Data":"24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102"}
Jul 29 11:02:53.762540 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.762513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" event={"ID":"b5eb6eb8-3fab-4327-9545-392ea1d79418","Type":"ContainerStarted","Data":"68fdbe388c68c448b7e61fdb86af6a0b9aa61b133e857e9fde9518a0041cd22a"}
Jul 29 11:02:53.781807 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:53.781753    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" podStartSLOduration=0.781739644 podStartE2EDuration="781.739644ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:02:53.779028863 +0000 UTC m=+2049.959410742" watchObservedRunningTime="2026-07-29 11:02:53.781739644 +0000 UTC m=+2049.962121523"
Jul 29 11:02:54.344166 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:02:54.344141    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:03:03.343818 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:03:03.343774    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:03:03.356428 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:03:03.356400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:03:03.825381 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:03:03.825350    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:03:46.128054 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:03:46.127950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:03:46.132162 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:03:46.132140    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:04:32.994579 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:32.994551    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:04:33.003699 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:04:33.003664    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:04:33.003699 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:33.003699 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:33.003699 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102" cmd=["/bin/sleep","15"]
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.003720    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" message="" err=<
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:04:33.003738    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerName="main" containerID="cri-o://24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102"
Jul 29 11:04:33.003989 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.003824    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerName="main" containerID="cri-o://24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102" gracePeriod=60
Jul 29 11:04:33.262864 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.262795    2574 generic.go:356] "Generic (PLEG): container finished" podID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerID="24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102" exitCode=0
Jul 29 11:04:33.262990 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.262868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" event={"ID":"b5eb6eb8-3fab-4327-9545-392ea1d79418","Type":"ContainerDied","Data":"24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102"}
Jul 29 11:04:33.354754 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.354730    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:04:33.482852 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482829    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483006 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482868    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483006 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96ml8\" (UniqueName: \"kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483006 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482954    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483006 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483006 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.482992    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm\") pod \"b5eb6eb8-3fab-4327-9545-392ea1d79418\" (UID: \"b5eb6eb8-3fab-4327-9545-392ea1d79418\") "
Jul 29 11:04:33.483313 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483119    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home" (OuterVolumeSpecName: "home") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:33.483313 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483255    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:33.483313 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483263    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:33.483468 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483445    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:33.483521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483473    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:33.483521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.483488    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:33.485417 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.485391    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm" (OuterVolumeSpecName: "dshm") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:04:33.485417 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.485406    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8" (OuterVolumeSpecName: "kube-api-access-96ml8") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "kube-api-access-96ml8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:04:33.485417 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.485406    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5eb6eb8-3fab-4327-9545-392ea1d79418" (UID: "b5eb6eb8-3fab-4327-9545-392ea1d79418"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:04:33.583968 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.583948    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5eb6eb8-3fab-4327-9545-392ea1d79418-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:33.583968 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.583970    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96ml8\" (UniqueName: \"kubernetes.io/projected/b5eb6eb8-3fab-4327-9545-392ea1d79418-kube-api-access-96ml8\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:33.584117 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:33.583979    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5eb6eb8-3fab-4327-9545-392ea1d79418-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:04:34.270142 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.270101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr" event={"ID":"b5eb6eb8-3fab-4327-9545-392ea1d79418","Type":"ContainerDied","Data":"68fdbe388c68c448b7e61fdb86af6a0b9aa61b133e857e9fde9518a0041cd22a"}
Jul 29 11:04:34.270142 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.270134    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"
Jul 29 11:04:34.270615 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.270180    2574 scope.go:117] "RemoveContainer" containerID="24d4781e66a765fce6ace2ea7db97e41cb4325d74c3db125255c89aee9d6a102"
Jul 29 11:04:34.297521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.297496    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:04:34.299446 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.299426    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf8679696mxdr"]
Jul 29 11:04:34.305676 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:04:34.305656    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eb6eb8_3fab_4327_9545_392ea1d79418.slice\": RecentStats: unable to find data in memory cache]"
Jul 29 11:04:34.342053 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:34.342029    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" path="/var/lib/kubelet/pods/b5eb6eb8-3fab-4327-9545-392ea1d79418/volumes"
Jul 29 11:04:57.619692 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.619654    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:04:57.620133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.620117    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerName="main"
Jul 29 11:04:57.620176 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.620134    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerName="main"
Jul 29 11:04:57.620274 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.620259    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5eb6eb8-3fab-4327-9545-392ea1d79418" containerName="main"
Jul 29 11:04:57.623617 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.623598    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.625473 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625453    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 29 11:04:57.625691 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625672    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 29 11:04:57.625844 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625822    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:04:57.625916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625862    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 29 11:04:57.625916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625871    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-8lpxb\""
Jul 29 11:04:57.625916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.625880    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 29 11:04:57.635053 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.635032    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:04:57.655914 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.655886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.655914 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.655928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczbp\" (UniqueName: \"kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656096 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.655990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656096 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.656016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656319 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.656301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656383 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.656340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656383 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.656367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.656489 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.656382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757354 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757750 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczbp\" (UniqueName: \"kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757750 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757750 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757929 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.757929 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757886    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.758039 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.757966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.758099 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.758073    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.758185 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.758168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.759985 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.759965    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.760348 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.760330    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.765688 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.765665    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczbp\" (UniqueName: \"kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:57.935133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:57.935070    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:04:58.076309 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:58.076285    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:04:58.078366 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:04:58.078332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1608ac8_5e3f_44ae_b9cb_aedad126d14e.slice/crio-a4bca1ae0ea3986fc7e44d4dd40ce8112b3eefe7ccb1ce83c5f7e9d032046564 WatchSource:0}: Error finding container a4bca1ae0ea3986fc7e44d4dd40ce8112b3eefe7ccb1ce83c5f7e9d032046564: Status 404 returned error can't find the container with id a4bca1ae0ea3986fc7e44d4dd40ce8112b3eefe7ccb1ce83c5f7e9d032046564
Jul 29 11:04:58.397869 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:58.397834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerStarted","Data":"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"}
Jul 29 11:04:58.398051 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:04:58.397891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerStarted","Data":"a4bca1ae0ea3986fc7e44d4dd40ce8112b3eefe7ccb1ce83c5f7e9d032046564"}
Jul 29 11:05:03.424523 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:03.424494    2574 generic.go:356] "Generic (PLEG): container finished" podID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerID="82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a" exitCode=0
Jul 29 11:05:03.424897 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:03.424538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerDied","Data":"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"}
Jul 29 11:05:04.431466 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:04.431423    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerStarted","Data":"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"}
Jul 29 11:05:04.454703 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:04.454641    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podStartSLOduration=3.141155783 podStartE2EDuration="7.454624403s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.31346862s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:05:04.452152899 +0000 UTC m=+2180.632534779" watchObservedRunningTime="2026-07-29 11:05:04.454624403 +0000 UTC m=+2180.635006282"
Jul 29 11:05:04.935515 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:04.935484    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:05:07.935481 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:07.935445    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:05:07.937037 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:07.937010    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:05:17.936150 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:17.936106    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:05:27.936017 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:27.935974    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:05:37.935638 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:37.935597    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:05:47.935525 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:47.935483    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:05:57.936272 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:05:57.936211    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:06:07.936310 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:07.936192    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Jul 29 11:06:17.946553 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:17.946521    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:06:17.954416 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:17.954390    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:06:42.176410 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:42.176374    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:06:57.202027 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.201959    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main" containerID="cri-o://2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42" gracePeriod=45
Jul 29 11:06:57.850248 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.850211    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:06:57.934015 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.933947    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934015 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.933985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934015    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934061    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934095    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934121    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934177    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jczbp\" (UniqueName: \"kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934491 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934212    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home\") pod \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\" (UID: \"a1608ac8-5e3f-44ae-b9cb-aedad126d14e\") "
Jul 29 11:06:57.934491 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934434    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:57.934491 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934469    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:06:57.934653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934560    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:57.934653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934572    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-cabundle-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:57.934837 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.934799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home" (OuterVolumeSpecName: "home") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:57.936695 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.936670    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm" (OuterVolumeSpecName: "dshm") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:57.936788 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.936769    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp" (OuterVolumeSpecName: "kube-api-access-jczbp") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "kube-api-access-jczbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:06:57.936881 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.936864    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:06:57.946612 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.946589    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:57.993203 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:57.993176    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1608ac8-5e3f-44ae-b9cb-aedad126d14e" (UID: "a1608ac8-5e3f-44ae-b9cb-aedad126d14e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:06:58.030816 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.030792    2574 generic.go:356] "Generic (PLEG): container finished" podID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerID="2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42" exitCode=0
Jul 29 11:06:58.030902 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.030848    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"
Jul 29 11:06:58.030902 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.030852    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerDied","Data":"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"}
Jul 29 11:06:58.030999 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.030932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj" event={"ID":"a1608ac8-5e3f-44ae-b9cb-aedad126d14e","Type":"ContainerDied","Data":"a4bca1ae0ea3986fc7e44d4dd40ce8112b3eefe7ccb1ce83c5f7e9d032046564"}
Jul 29 11:06:58.030999 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.030956    2574 scope.go:117] "RemoveContainer" containerID="2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"
Jul 29 11:06:58.034888 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034868    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.034983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034893    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.034983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034905    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.034983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034913    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.034983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034920    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jczbp\" (UniqueName: \"kubernetes.io/projected/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-kube-api-access-jczbp\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.034983 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.034927    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1608ac8-5e3f-44ae-b9cb-aedad126d14e-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:06:58.039898 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.039881    2574 scope.go:117] "RemoveContainer" containerID="82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"
Jul 29 11:06:58.048306 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.048291    2574 scope.go:117] "RemoveContainer" containerID="2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"
Jul 29 11:06:58.048551 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:06:58.048528    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42\": container with ID starting with 2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42 not found: ID does not exist" containerID="2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"
Jul 29 11:06:58.048627 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.048561    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42"} err="failed to get container status \"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42\": rpc error: code = NotFound desc = could not find container \"2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42\": container with ID starting with 2218ba7e209834a9aec830c7d29deab5fb90545794948a4fca15736db5302b42 not found: ID does not exist"
Jul 29 11:06:58.048627 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.048583    2574 scope.go:117] "RemoveContainer" containerID="82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"
Jul 29 11:06:58.048798 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:06:58.048783    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a\": container with ID starting with 82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a not found: ID does not exist" containerID="82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"
Jul 29 11:06:58.048850 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.048806    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a"} err="failed to get container status \"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a\": rpc error: code = NotFound desc = could not find container \"82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a\": container with ID starting with 82b5e3cbb92e4f5f66bdf19b430587a9910beb3ac1e7f8ff475f487c4f73de5a not found: ID does not exist"
Jul 29 11:06:58.052059 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.052036    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:06:58.055557 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.055538    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-776c5786b-tkbwj"]
Jul 29 11:06:58.342787 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:06:58.342761    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" path="/var/lib/kubelet/pods/a1608ac8-5e3f-44ae-b9cb-aedad126d14e/volumes"
Jul 29 11:07:12.225370 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225340    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:07:12.225830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225669    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main"
Jul 29 11:07:12.225830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225680    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main"
Jul 29 11:07:12.225830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225689    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="storage-initializer"
Jul 29 11:07:12.225830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225695    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="storage-initializer"
Jul 29 11:07:12.225830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.225744    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1608ac8-5e3f-44ae-b9cb-aedad126d14e" containerName="main"
Jul 29 11:07:12.230622 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.230601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.232571 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.232547    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 29 11:07:12.232713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.232623    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-4b5bz\""
Jul 29 11:07:12.232713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.232557    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 29 11:07:12.232713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.232693    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 29 11:07:12.232887 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.232779    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 29 11:07:12.233028 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.233010    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:07:12.240765 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.240743    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:07:12.337014 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.336985    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337014 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmdj\" (UniqueName: \"kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.337521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.337258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438482 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438456    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438602 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438487    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438602 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438602 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmdj\" (UniqueName: \"kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.438795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.439013 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438863    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.439013 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.438881    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.439472 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.439450    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.439580 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.439472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.439580 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.439477    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.440937 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.440917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.441206 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.441188    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.446053 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.446032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmdj\" (UniqueName: \"kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.542682 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.542656    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:12.680470 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:12.680442    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:07:12.681958 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:07:12.681933    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518eafaf_2107_4546_8834_0b47d0754ffe.slice/crio-b1b29ea238b7745681a3376e3adef7361f9cb078a24c95427ec9dff9c5e62fb9 WatchSource:0}: Error finding container b1b29ea238b7745681a3376e3adef7361f9cb078a24c95427ec9dff9c5e62fb9: Status 404 returned error can't find the container with id b1b29ea238b7745681a3376e3adef7361f9cb078a24c95427ec9dff9c5e62fb9
Jul 29 11:07:13.111047 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:13.111007    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerStarted","Data":"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"}
Jul 29 11:07:13.111207 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:13.111066    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerStarted","Data":"b1b29ea238b7745681a3376e3adef7361f9cb078a24c95427ec9dff9c5e62fb9"}
Jul 29 11:07:21.151910 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:21.151873    2574 generic.go:356] "Generic (PLEG): container finished" podID="518eafaf-2107-4546-8834-0b47d0754ffe" containerID="b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61" exitCode=0
Jul 29 11:07:21.152339 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:21.151955    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerDied","Data":"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"}
Jul 29 11:07:22.160968 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:22.160926    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerStarted","Data":"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"}
Jul 29 11:07:22.180352 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:22.180307    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podStartSLOduration=2.197160964 podStartE2EDuration="10.180293428s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.983132464s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:07:22.178685174 +0000 UTC m=+2318.359067055" watchObservedRunningTime="2026-07-29 11:07:22.180293428 +0000 UTC m=+2318.360675310"
Jul 29 11:07:22.543044 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:22.543009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:22.543257 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:22.543065    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:07:22.544927 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:22.544899    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:07:32.543413 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:32.543376    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:07:42.543825 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:42.543744    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:07:52.543314 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:07:52.543270    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:08:02.543615 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:02.543579    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:08:12.543722 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:12.543677    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:08:22.543465 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:22.543416    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": dial tcp 10.132.0.57:8000: connect: connection refused"
Jul 29 11:08:32.553825 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:32.553782    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:08:32.562093 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:32.562065    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:08:33.805302 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:33.805270    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:08:46.161132 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:46.161014    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:08:46.168128 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:46.168107    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:08:49.566758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:49.566714    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main" containerID="cri-o://368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a" gracePeriod=45
Jul 29 11:08:50.128128 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.128105    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:08:50.157019 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.156995    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157158 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157028    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157158 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157065    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157158 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157158 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157136    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157384 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157284    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157384 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157377    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157486 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157419    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkmdj\" (UniqueName: \"kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj\") pod \"518eafaf-2107-4546-8834-0b47d0754ffe\" (UID: \"518eafaf-2107-4546-8834-0b47d0754ffe\") "
Jul 29 11:08:50.157575 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache" (OuterVolumeSpecName: "model-cache") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:50.157703 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157596    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:50.157771 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157734    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.157771 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157753    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.157875 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157761    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:08:50.157875 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.157859    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home" (OuterVolumeSpecName: "home") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:50.159710 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.159680    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm" (OuterVolumeSpecName: "dshm") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:50.160009 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.159978    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:08:50.160111 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.160015    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj" (OuterVolumeSpecName: "kube-api-access-rkmdj") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "kube-api-access-rkmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:08:50.219804 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.219730    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "518eafaf-2107-4546-8834-0b47d0754ffe" (UID: "518eafaf-2107-4546-8834-0b47d0754ffe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:08:50.259117 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259090    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkmdj\" (UniqueName: \"kubernetes.io/projected/518eafaf-2107-4546-8834-0b47d0754ffe-kube-api-access-rkmdj\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.259117 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259115    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.259296 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259124    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518eafaf-2107-4546-8834-0b47d0754ffe-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.259296 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259133    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.259296 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259141    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518eafaf-2107-4546-8834-0b47d0754ffe-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.259296 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.259149    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/518eafaf-2107-4546-8834-0b47d0754ffe-cabundle-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:08:50.622304 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.622267    2574 generic.go:356] "Generic (PLEG): container finished" podID="518eafaf-2107-4546-8834-0b47d0754ffe" containerID="368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a" exitCode=0
Jul 29 11:08:50.622778 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.622337    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"
Jul 29 11:08:50.622778 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.622356    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerDied","Data":"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"}
Jul 29 11:08:50.622778 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.622456    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv" event={"ID":"518eafaf-2107-4546-8834-0b47d0754ffe","Type":"ContainerDied","Data":"b1b29ea238b7745681a3376e3adef7361f9cb078a24c95427ec9dff9c5e62fb9"}
Jul 29 11:08:50.622778 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.622488    2574 scope.go:117] "RemoveContainer" containerID="368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"
Jul 29 11:08:50.631624 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.631609    2574 scope.go:117] "RemoveContainer" containerID="b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"
Jul 29 11:08:50.641202 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.641129    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:08:50.641354 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.641332    2574 scope.go:117] "RemoveContainer" containerID="368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"
Jul 29 11:08:50.641641 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:08:50.641614    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a\": container with ID starting with 368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a not found: ID does not exist" containerID="368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"
Jul 29 11:08:50.641725 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.641639    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a"} err="failed to get container status \"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a\": rpc error: code = NotFound desc = could not find container \"368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a\": container with ID starting with 368f7a7e3c2ca53042a1891ce6d305f3c28b7b4449a9e21e5764e0fb54e5336a not found: ID does not exist"
Jul 29 11:08:50.641725 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.641656    2574 scope.go:117] "RemoveContainer" containerID="b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"
Jul 29 11:08:50.641867 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:08:50.641852    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61\": container with ID starting with b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61 not found: ID does not exist" containerID="b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"
Jul 29 11:08:50.641924 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.641870    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61"} err="failed to get container status \"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61\": rpc error: code = NotFound desc = could not find container \"b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61\": container with ID starting with b2418cd6c5d2cd1d25ab8b1286721bca325405c2184c026ee107a26a91a5fe61 not found: ID does not exist"
Jul 29 11:08:50.643367 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:50.643347    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-75c67cd687-9qtsv"]
Jul 29 11:08:52.342090 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:52.342057    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" path="/var/lib/kubelet/pods/518eafaf-2107-4546-8834-0b47d0754ffe/volumes"
Jul 29 11:08:57.100624 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100594    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"]
Jul 29 11:08:57.101023 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100898    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="storage-initializer"
Jul 29 11:08:57.101023 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100909    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="storage-initializer"
Jul 29 11:08:57.101023 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100920    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main"
Jul 29 11:08:57.101023 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100926    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main"
Jul 29 11:08:57.101023 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.100990    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="518eafaf-2107-4546-8834-0b47d0754ffe" containerName="main"
Jul 29 11:08:57.105398 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.105381    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:08:57.107960 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.107936    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 11:08:57.113270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.108919    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 11:08:57.113270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.109807    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-rqgbm\""
Jul 29 11:08:57.113270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.111198    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"]
Jul 29 11:08:57.216442 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.216415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jzm\" (UniqueName: \"kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:08:57.317027 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.316990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jzm\" (UniqueName: \"kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:08:57.325678 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:08:57.325650    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jzm\" (UniqueName: \"kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:03.672559 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:03.672521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:03.676799 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:03.676767    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:03.773498 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:03.773466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:03.773683 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:03.773628    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:03.776164 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:03.776144    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") DevicePath \"csi-062d159357c9ec423e4091170604cdacdfc5acf45d2bc7cc42e634ad31c3db08\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:04.081341 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.081311    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b908f2c210e1f7582e0165bcb997761ab451fabaf97c4341effce849b347c50c/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:04.095312 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.095285    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"e2e-pvc-model-download-2xk7l\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:04.321148 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.321052    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:04.455784 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.455757    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"]
Jul 29 11:09:04.458408 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:09:04.458369    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66bbd5e2_0c03_4059_9e03_bd001c130beb.slice/crio-539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af WatchSource:0}: Error finding container 539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af: Status 404 returned error can't find the container with id 539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af
Jul 29 11:09:04.692015 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.691918    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l" event={"ID":"66bbd5e2-0c03-4059-9e03-bd001c130beb","Type":"ContainerStarted","Data":"0827e771d4ec0365672fc4194420a195de9210c3c71a9932fc65d21f689e3f8b"}
Jul 29 11:09:04.692015 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:04.691963    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l" event={"ID":"66bbd5e2-0c03-4059-9e03-bd001c130beb","Type":"ContainerStarted","Data":"539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af"}
Jul 29 11:09:11.728732 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:11.728694    2574 generic.go:356] "Generic (PLEG): container finished" podID="66bbd5e2-0c03-4059-9e03-bd001c130beb" containerID="0827e771d4ec0365672fc4194420a195de9210c3c71a9932fc65d21f689e3f8b" exitCode=0
Jul 29 11:09:11.729125 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:11.728761    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l" event={"ID":"66bbd5e2-0c03-4059-9e03-bd001c130beb","Type":"ContainerDied","Data":"0827e771d4ec0365672fc4194420a195de9210c3c71a9932fc65d21f689e3f8b"}
Jul 29 11:09:12.889059 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:12.889037    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:12.946960 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:12.946937    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66jzm\" (UniqueName: \"kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm\") pod \"66bbd5e2-0c03-4059-9e03-bd001c130beb\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") "
Jul 29 11:09:12.947121 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:12.947039    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"66bbd5e2-0c03-4059-9e03-bd001c130beb\" (UID: \"66bbd5e2-0c03-4059-9e03-bd001c130beb\") "
Jul 29 11:09:12.949321 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:12.949293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm" (OuterVolumeSpecName: "kube-api-access-66jzm") pod "66bbd5e2-0c03-4059-9e03-bd001c130beb" (UID: "66bbd5e2-0c03-4059-9e03-bd001c130beb"). InnerVolumeSpecName "kube-api-access-66jzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:09:12.951022 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:12.950997    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36" (OuterVolumeSpecName: "model-storage") pod "66bbd5e2-0c03-4059-9e03-bd001c130beb" (UID: "66bbd5e2-0c03-4059-9e03-bd001c130beb"). InnerVolumeSpecName "pvc-570d44a8-78ac-479c-ba80-b11eee07e086". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 11:09:13.048115 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.048093    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jzm\" (UniqueName: \"kubernetes.io/projected/66bbd5e2-0c03-4059-9e03-bd001c130beb-kube-api-access-66jzm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:09:13.048205 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.048135    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") on node \"ip-10-0-133-86.ec2.internal\" "
Jul 29 11:09:13.160123 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.160104    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-570d44a8-78ac-479c-ba80-b11eee07e086" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36") on node "ip-10-0-133-86.ec2.internal"
Jul 29 11:09:13.249622 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.249599    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"csi-062d159357c9ec423e4091170604cdacdfc5acf45d2bc7cc42e634ad31c3db08\""
Jul 29 11:09:13.739788 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.739754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l" event={"ID":"66bbd5e2-0c03-4059-9e03-bd001c130beb","Type":"ContainerDied","Data":"539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af"}
Jul 29 11:09:13.739788 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.739790    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539744604ce4b2b2df07f26c2f8f479c654c7a53d3b3289ebbd4a12d8bd1f2af"
Jul 29 11:09:13.739788 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:13.739764    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"
Jul 29 11:09:21.226595 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.226540    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:09:21.227042 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.227019    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bbd5e2-0c03-4059-9e03-bd001c130beb" containerName="storage-initializer"
Jul 29 11:09:21.227042 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.227037    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bbd5e2-0c03-4059-9e03-bd001c130beb" containerName="storage-initializer"
Jul 29 11:09:21.227161 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.227148    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bbd5e2-0c03-4059-9e03-bd001c130beb" containerName="storage-initializer"
Jul 29 11:09:21.230033 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.230013    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.232325 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.232301    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-rqgbm\""
Jul 29 11:09:21.232505 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.232482    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 29 11:09:21.232505 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.232490    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 29 11:09:21.232707 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.232503    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 29 11:09:21.240364 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.240344    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:09:21.310837 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.310943 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.310943 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.310943 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.311075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.311075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.310982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411471 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411589 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411589 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411589 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411589 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411875 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411937 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.411937 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.411930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.412080 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.412057    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.413890 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.413870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.414162 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.414144    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:21.418752 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:21.418731    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.651103 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.651071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.655521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.655490    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.752409 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.752380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.752513 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.752497    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.754526 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.754505    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") DevicePath \"csi-062d159357c9ec423e4091170604cdacdfc5acf45d2bc7cc42e634ad31c3db08\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.845269 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.845240    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b908f2c210e1f7582e0165bcb997761ab451fabaf97c4341effce849b347c50c/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:25.859110 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:25.859084    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:26.041262 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:26.041238    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:26.177489 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:26.177462    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:09:26.179398 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:09:26.179372    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84700e7b_156d_4314_888d_76dde4bef973.slice/crio-b9a75cb70bde32c87aa04150ff1357fb3b93a2df14da53f948b783a8971c41f1 WatchSource:0}: Error finding container b9a75cb70bde32c87aa04150ff1357fb3b93a2df14da53f948b783a8971c41f1: Status 404 returned error can't find the container with id b9a75cb70bde32c87aa04150ff1357fb3b93a2df14da53f948b783a8971c41f1
Jul 29 11:09:26.810412 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:26.810376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" event={"ID":"84700e7b-156d-4314-888d-76dde4bef973","Type":"ContainerStarted","Data":"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"}
Jul 29 11:09:26.810801 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:26.810437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" event={"ID":"84700e7b-156d-4314-888d-76dde4bef973","Type":"ContainerStarted","Data":"b9a75cb70bde32c87aa04150ff1357fb3b93a2df14da53f948b783a8971c41f1"}
Jul 29 11:09:27.041511 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:27.041478    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:36.042469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:36.042432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:09:36.043784 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:36.043754    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:09:46.042643 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:46.042601    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:09:56.042647 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:09:56.042603    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:06.041981 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:06.041935    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:16.042564 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:16.042522    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:26.042230 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:26.042185    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:36.042155 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:36.042064    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:46.042475 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:46.042428    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:10:56.042525 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:10:56.042470    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:11:06.042056 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:06.042006    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": dial tcp 10.132.0.59:8000: connect: connection refused"
Jul 29 11:11:16.052121 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:16.052082    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:11:16.059893 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:16.059862    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:11:16.071258 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:16.071196    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podStartSLOduration=115.071182215 podStartE2EDuration="1m55.071182215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:09:26.83181404 +0000 UTC m=+2443.012195923" watchObservedRunningTime="2026-07-29 11:11:16.071182215 +0000 UTC m=+2552.251564094"
Jul 29 11:11:21.114270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:21.114237    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:11:36.138994 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.138956    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main" containerID="cri-o://7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651" gracePeriod=45
Jul 29 11:11:36.819565 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.819535    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:11:36.890882 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.890853    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891010 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.890920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891010 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.890942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891017    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891037    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891086    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m\") pod \"84700e7b-156d-4314-888d-76dde4bef973\" (UID: \"84700e7b-156d-4314-888d-76dde4bef973\") "
Jul 29 11:11:36.891109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache" (OuterVolumeSpecName: "model-cache") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:11:36.891416 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891362    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.891917 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.891848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home" (OuterVolumeSpecName: "home") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:11:36.893392 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.893352    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:11:36.894002 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.893977    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm" (OuterVolumeSpecName: "dshm") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:11:36.894086 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.894016    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m" (OuterVolumeSpecName: "kube-api-access-ctq2m") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "kube-api-access-ctq2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:11:36.894246 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.894205    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36" (OuterVolumeSpecName: "kserve-pvc-source") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "pvc-570d44a8-78ac-479c-ba80-b11eee07e086". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 11:11:36.902568 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.902543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84700e7b-156d-4314-888d-76dde4bef973" (UID: "84700e7b-156d-4314-888d-76dde4bef973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:11:36.992403 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992381    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.992403 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992400    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctq2m\" (UniqueName: \"kubernetes.io/projected/84700e7b-156d-4314-888d-76dde4bef973-kube-api-access-ctq2m\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.992530 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992412    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84700e7b-156d-4314-888d-76dde4bef973-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.992530 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992421    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:36.992530 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992444    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") on node \"ip-10-0-133-86.ec2.internal\" "
Jul 29 11:11:36.992530 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:36.992452    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84700e7b-156d-4314-888d-76dde4bef973-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:37.018799 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.018784    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-570d44a8-78ac-479c-ba80-b11eee07e086" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36") on node "ip-10-0-133-86.ec2.internal"
Jul 29 11:11:37.093143 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.093119    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-570d44a8-78ac-479c-ba80-b11eee07e086\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0b1483c48739ade36\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"csi-062d159357c9ec423e4091170604cdacdfc5acf45d2bc7cc42e634ad31c3db08\""
Jul 29 11:11:37.478740 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.478702    2574 generic.go:356] "Generic (PLEG): container finished" podID="84700e7b-156d-4314-888d-76dde4bef973" containerID="7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651" exitCode=0
Jul 29 11:11:37.479119 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.478747    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" event={"ID":"84700e7b-156d-4314-888d-76dde4bef973","Type":"ContainerDied","Data":"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"}
Jul 29 11:11:37.479119 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.478758    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"
Jul 29 11:11:37.479119 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.478798    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv" event={"ID":"84700e7b-156d-4314-888d-76dde4bef973","Type":"ContainerDied","Data":"b9a75cb70bde32c87aa04150ff1357fb3b93a2df14da53f948b783a8971c41f1"}
Jul 29 11:11:37.479119 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.478827    2574 scope.go:117] "RemoveContainer" containerID="7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"
Jul 29 11:11:37.489048 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.489030    2574 scope.go:117] "RemoveContainer" containerID="7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"
Jul 29 11:11:37.489344 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:11:37.489315    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651\": container with ID starting with 7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651 not found: ID does not exist" containerID="7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"
Jul 29 11:11:37.489454 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.489349    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651"} err="failed to get container status \"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651\": rpc error: code = NotFound desc = could not find container \"7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651\": container with ID starting with 7170c8837fd4170371460be5569494b861ed391bdf52a9be359a40d7850c3651 not found: ID does not exist"
Jul 29 11:11:37.500277 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.500253    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:11:37.503352 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:37.503331    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcdnwsv"]
Jul 29 11:11:38.342453 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:38.342418    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84700e7b-156d-4314-888d-76dde4bef973" path="/var/lib/kubelet/pods/84700e7b-156d-4314-888d-76dde4bef973/volumes"
Jul 29 11:11:44.336001 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.335962    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"]
Jul 29 11:11:44.336423 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.336326    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main"
Jul 29 11:11:44.336423 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.336339    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main"
Jul 29 11:11:44.336423 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.336403    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84700e7b-156d-4314-888d-76dde4bef973" containerName="main"
Jul 29 11:11:44.341009 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.340971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:44.343851 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.343821    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 11:11:44.344442 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.344406    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 11:11:44.344569 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.344406    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-mcnx5\""
Jul 29 11:11:44.348157 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.348130    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"]
Jul 29 11:11:44.444391 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.444363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9w6j\" (UniqueName: \"kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:44.545114 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.545079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9w6j\" (UniqueName: \"kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:44.553068 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:44.553044    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9w6j\" (UniqueName: \"kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:46.084444 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:46.084194    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"]
Jul 29 11:11:46.087009 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:46.086985    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-2xk7l"]
Jul 29 11:11:46.347744 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:46.347667    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bbd5e2-0c03-4059-9e03-bd001c130beb" path="/var/lib/kubelet/pods/66bbd5e2-0c03-4059-9e03-bd001c130beb/volumes"
Jul 29 11:11:48.171692 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.171616    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.175834 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.175812    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.272096 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.272067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.272251 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.272171    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.274404 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.274385    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") DevicePath \"csi-8ff94c8e73e106c66a59b8f82c20c6111f6926f6cfa82df1e31b0a6ce37431b1\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.527823 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.527794    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/73455ade1b2cb3965604e4275140c5271fd0f1a79a71679e0fd4fd7adbd736dd/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.542092 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.542065    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"e2e-pvc-model-download-hcdhx\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.553009 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.552988    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:48.681817 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:48.681785    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"]
Jul 29 11:11:48.684293 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:11:48.684266    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93495d2b_b97d_498f_b0ee_8a27962fd428.slice/crio-7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418 WatchSource:0}: Error finding container 7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418: Status 404 returned error can't find the container with id 7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418
Jul 29 11:11:49.543403 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:49.543362    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx" event={"ID":"93495d2b-b97d-498f-b0ee-8a27962fd428","Type":"ContainerStarted","Data":"a5857a1d7135efabe8202a139f1891b033a56d3d3d73fd5db6bd884197404195"}
Jul 29 11:11:49.543403 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:49.543413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx" event={"ID":"93495d2b-b97d-498f-b0ee-8a27962fd428","Type":"ContainerStarted","Data":"7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418"}
Jul 29 11:11:49.556262 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:49.556202    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx" podStartSLOduration=8.556180332 podStartE2EDuration="8.556180332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:11:49.556073639 +0000 UTC m=+2585.736455519" watchObservedRunningTime="2026-07-29 11:11:49.556180332 +0000 UTC m=+2585.736562214"
Jul 29 11:11:55.582956 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:55.582928    2574 generic.go:356] "Generic (PLEG): container finished" podID="93495d2b-b97d-498f-b0ee-8a27962fd428" containerID="a5857a1d7135efabe8202a139f1891b033a56d3d3d73fd5db6bd884197404195" exitCode=0
Jul 29 11:11:55.583380 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:55.583003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx" event={"ID":"93495d2b-b97d-498f-b0ee-8a27962fd428","Type":"ContainerDied","Data":"a5857a1d7135efabe8202a139f1891b033a56d3d3d73fd5db6bd884197404195"}
Jul 29 11:11:56.736426 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.736402    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:56.840005 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.839948    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"93495d2b-b97d-498f-b0ee-8a27962fd428\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") "
Jul 29 11:11:56.840118 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.840014    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9w6j\" (UniqueName: \"kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j\") pod \"93495d2b-b97d-498f-b0ee-8a27962fd428\" (UID: \"93495d2b-b97d-498f-b0ee-8a27962fd428\") "
Jul 29 11:11:56.842577 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.842545    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j" (OuterVolumeSpecName: "kube-api-access-c9w6j") pod "93495d2b-b97d-498f-b0ee-8a27962fd428" (UID: "93495d2b-b97d-498f-b0ee-8a27962fd428"). InnerVolumeSpecName "kube-api-access-c9w6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:11:56.842945 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.842926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8" (OuterVolumeSpecName: "model-storage") pod "93495d2b-b97d-498f-b0ee-8a27962fd428" (UID: "93495d2b-b97d-498f-b0ee-8a27962fd428"). InnerVolumeSpecName "pvc-ce3748df-ddb8-4bce-a255-9229c58859db". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 29 11:11:56.940830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.940796    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9w6j\" (UniqueName: \"kubernetes.io/projected/93495d2b-b97d-498f-b0ee-8a27962fd428-kube-api-access-c9w6j\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:11:56.940969 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:56.940843    2574 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") on node \"ip-10-0-133-86.ec2.internal\" "
Jul 29 11:11:57.054688 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:57.054670    2574 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-ce3748df-ddb8-4bce-a255-9229c58859db" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8") on node "ip-10-0-133-86.ec2.internal"
Jul 29 11:11:57.142563 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:57.142511    2574 reconciler_common.go:299] "Volume detached for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"csi-8ff94c8e73e106c66a59b8f82c20c6111f6926f6cfa82df1e31b0a6ce37431b1\""
Jul 29 11:11:57.596313 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:57.596283    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"
Jul 29 11:11:57.596472 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:57.596318    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx" event={"ID":"93495d2b-b97d-498f-b0ee-8a27962fd428","Type":"ContainerDied","Data":"7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418"}
Jul 29 11:11:57.596472 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:11:57.596356    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d4a5ef14ffde0e25c90ed17fdc59f7862f96879e2b9deed9ae52172357c4418"
Jul 29 11:12:03.863030 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.862993    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"]
Jul 29 11:12:03.863731 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.863455    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93495d2b-b97d-498f-b0ee-8a27962fd428" containerName="storage-initializer"
Jul 29 11:12:03.863731 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.863474    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="93495d2b-b97d-498f-b0ee-8a27962fd428" containerName="storage-initializer"
Jul 29 11:12:03.863731 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.863568    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="93495d2b-b97d-498f-b0ee-8a27962fd428" containerName="storage-initializer"
Jul 29 11:12:03.866925 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.866902    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.869099 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.869075    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 29 11:12:03.869272 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.869246    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-mcnx5\""
Jul 29 11:12:03.869393 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.869303    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 29 11:12:03.869393 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.869081    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-b9nll\""
Jul 29 11:12:03.869393 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.869097    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 29 11:12:03.883412 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.883386    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"]
Jul 29 11:12:03.893727 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.893677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.893850 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.893762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.893946 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.893913    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/b3e34fe2-65ed-4035-b463-759754ab7e02-kube-api-access-lvp74\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.894077 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.893973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.894077 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.894015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e34fe2-65ed-4035-b463-759754ab7e02-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.894077 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.894058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995054 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995180 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995071    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995180 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995106    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/b3e34fe2-65ed-4035-b463-759754ab7e02-kube-api-access-lvp74\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995180 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995180 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e34fe2-65ed-4035-b463-759754ab7e02-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995432 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995496    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995577 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.995577 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.995571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.997551 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.997529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3e34fe2-65ed-4035-b463-759754ab7e02-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:03.997809 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:03.997790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3e34fe2-65ed-4035-b463-759754ab7e02-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:04.001899 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:04.001877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvp74\" (UniqueName: \"kubernetes.io/projected/b3e34fe2-65ed-4035-b463-759754ab7e02-kube-api-access-lvp74\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.324042 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.323961    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.328970 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.328949    2574 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.424746 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.424721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.424878 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.424859    2574 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.426855 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.426836    2574 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") DevicePath \"csi-8ff94c8e73e106c66a59b8f82c20c6111f6926f6cfa82df1e31b0a6ce37431b1\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.519911 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.519876    2574 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/73455ade1b2cb3965604e4275140c5271fd0f1a79a71679e0fd4fd7adbd736dd/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.534856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.534837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-ce3748df-ddb8-4bce-a255-9229c58859db\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08e5dd0331e0563e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67\" (UID: \"b3e34fe2-65ed-4035-b463-759754ab7e02\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.779684 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.779658    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:07.917992 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:07.917963    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"]
Jul 29 11:12:07.919486 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:12:07.919461    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e34fe2_65ed_4035_b463_759754ab7e02.slice/crio-36338099daa7706600daef5ffb8b5ed0e0f8eb208eac8cda8cd7522bfc81c9dd WatchSource:0}: Error finding container 36338099daa7706600daef5ffb8b5ed0e0f8eb208eac8cda8cd7522bfc81c9dd: Status 404 returned error can't find the container with id 36338099daa7706600daef5ffb8b5ed0e0f8eb208eac8cda8cd7522bfc81c9dd
Jul 29 11:12:08.648571 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:08.648535    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" event={"ID":"b3e34fe2-65ed-4035-b463-759754ab7e02","Type":"ContainerStarted","Data":"35e8dddc73f5968744b03391a0f6b31ce1f5b556abb30bd1aecb24b97eb8ded9"}
Jul 29 11:12:08.648939 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:08.648595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" event={"ID":"b3e34fe2-65ed-4035-b463-759754ab7e02","Type":"ContainerStarted","Data":"36338099daa7706600daef5ffb8b5ed0e0f8eb208eac8cda8cd7522bfc81c9dd"}
Jul 29 11:12:08.648939 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:08.648625    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:09.655285 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:09.655238    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" event={"ID":"b3e34fe2-65ed-4035-b463-759754ab7e02","Type":"ContainerStarted","Data":"281b111b443a2afe9c4a30e2a385a18b609a255b96b0f8155aa8dd8883616784"}
Jul 29 11:12:09.676846 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:09.676791    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podStartSLOduration=6.676775204 podStartE2EDuration="6.676775204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 11:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:12:09.673526764 +0000 UTC m=+2605.853908625" watchObservedRunningTime="2026-07-29 11:12:09.676775204 +0000 UTC m=+2605.857157083"
Jul 29 11:12:17.780027 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:17.779989    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:17.780564 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:17.780044    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:17.781277 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:17.781243    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:12:17.793746 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:17.793727    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:12:27.780259 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:27.780191    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:12:37.781127 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:37.781080    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:12:47.780895 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:47.780842    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:12:57.780638 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:12:57.780590    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:07.780696 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:07.780650    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:17.780970 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:17.780918    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:27.780922 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:27.780871    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:37.780583 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:37.780535    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:46.198019 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:46.197908    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:13:46.203629 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:46.203610    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:13:47.780966 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:47.780919    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67" podUID="b3e34fe2-65ed-4035-b463-759754ab7e02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8001/health\": dial tcp 10.132.0.61:8001: connect: connection refused"
Jul 29 11:13:57.789739 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:57.789710    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:13:57.802813 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:13:57.802788    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67"
Jul 29 11:15:44.669669 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:15:44.669641    2574 scope.go:117] "RemoveContainer" containerID="0827e771d4ec0365672fc4194420a195de9210c3c71a9932fc65d21f689e3f8b"
Jul 29 11:18:46.230198 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:18:46.230098    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:18:46.237311 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:18:46.237292    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:23:46.263973 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:23:46.263954    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:23:46.275149 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:23:46.275125    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:24:36.100451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.100417    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:24:36.103793 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.103775    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.106413 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.106384    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-29cvz\""
Jul 29 11:24:36.106413 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.106411    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 29 11:24:36.106604 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.106512    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-nrqcq\""
Jul 29 11:24:36.106604 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.106522    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 29 11:24:36.106692 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.106620    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 29 11:24:36.114296 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.114275    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:24:36.256484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256450    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.256484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwj6f\" (UniqueName: \"kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.256689 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.256689 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.256689 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256579    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.256689 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.256633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.357949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.357883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.357949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.357909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.357949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.357933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.357949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.357952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.358207 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.358005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.358207 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.358022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwj6f\" (UniqueName: \"kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.358362 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.358258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.358421 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.358392    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.358468 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.358419    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.360390 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.360369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.360511 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.360493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.365330 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.365307    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwj6f\" (UniqueName: \"kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f\") pod \"autoscale-hpa-lws-kserve-mn-0-1\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.414830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.414810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:24:36.548392 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:36.548367    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:24:36.550606 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:24:36.550578    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8221e3_fa2e_46cf_9760_c042190da938.slice/crio-d69eb13b83411df3a49c1f42f4316daa10c68d6dd31e05328c6ce1ec5968c730 WatchSource:0}: Error finding container d69eb13b83411df3a49c1f42f4316daa10c68d6dd31e05328c6ce1ec5968c730: Status 404 returned error can't find the container with id d69eb13b83411df3a49c1f42f4316daa10c68d6dd31e05328c6ce1ec5968c730
Jul 29 11:24:37.408042 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:37.408010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"1d8221e3-fa2e-46cf-9760-c042190da938","Type":"ContainerStarted","Data":"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"}
Jul 29 11:24:37.408465 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:37.408054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"1d8221e3-fa2e-46cf-9760-c042190da938","Type":"ContainerStarted","Data":"d69eb13b83411df3a49c1f42f4316daa10c68d6dd31e05328c6ce1ec5968c730"}
Jul 29 11:24:37.429683 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:24:37.429641    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podStartSLOduration=1.429626216 podStartE2EDuration="1.429626216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:24:37.428692648 +0000 UTC m=+3353.609074529" watchObservedRunningTime="2026-07-29 11:24:37.429626216 +0000 UTC m=+3353.610008096"
Jul 29 11:27:02.519093 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:27:02.519061    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67_b3e34fe2-65ed-4035-b463-759754ab7e02/llm-d-routing-sidecar/0.log"
Jul 29 11:27:02.528019 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:27:02.528000    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56cn6j67_b3e34fe2-65ed-4035-b463-759754ab7e02/main/0.log"
Jul 29 11:28:06.907001 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.906945    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"]
Jul 29 11:28:06.910637 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.910616    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:06.913288 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.913253    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 29 11:28:06.913288 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.913271    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 29 11:28:06.913506 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.913308    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-5lcx4\""
Jul 29 11:28:06.913506 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.913398    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 29 11:28:06.913506 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.913308    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-7958q\""
Jul 29 11:28:06.919796 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.919771    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"]
Jul 29 11:28:06.978351 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.978327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh5j\" (UniqueName: \"kubernetes.io/projected/e9a55162-d774-4af1-9b73-1b1121a3eb6c-kube-api-access-bzh5j\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:06.978474 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:06.978403    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a55162-d774-4af1-9b73-1b1121a3eb6c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.079698 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.079671    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh5j\" (UniqueName: \"kubernetes.io/projected/e9a55162-d774-4af1-9b73-1b1121a3eb6c-kube-api-access-bzh5j\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.079824 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.079723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a55162-d774-4af1-9b73-1b1121a3eb6c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.082307 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.082289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a55162-d774-4af1-9b73-1b1121a3eb6c-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.087030 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.087004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh5j\" (UniqueName: \"kubernetes.io/projected/e9a55162-d774-4af1-9b73-1b1121a3eb6c-kube-api-access-bzh5j\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq\" (UID: \"e9a55162-d774-4af1-9b73-1b1121a3eb6c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.222653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.222596    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.354441 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.354341    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"]
Jul 29 11:28:07.356682 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:28:07.356656    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a55162_d774_4af1_9b73_1b1121a3eb6c.slice/crio-233a0c7bad0f6f3a9c4ffcb3b7cd884bd2ef8cdbbe29d6e228d189e99f1626f1 WatchSource:0}: Error finding container 233a0c7bad0f6f3a9c4ffcb3b7cd884bd2ef8cdbbe29d6e228d189e99f1626f1: Status 404 returned error can't find the container with id 233a0c7bad0f6f3a9c4ffcb3b7cd884bd2ef8cdbbe29d6e228d189e99f1626f1
Jul 29 11:28:07.454652 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.454614    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq" event={"ID":"e9a55162-d774-4af1-9b73-1b1121a3eb6c","Type":"ContainerStarted","Data":"2bc5422146ab7c6a535b045cd6de4b37b774e71117d99d3591d72df85c5987ee"}
Jul 29 11:28:07.454830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.454677    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq" event={"ID":"e9a55162-d774-4af1-9b73-1b1121a3eb6c","Type":"ContainerStarted","Data":"233a0c7bad0f6f3a9c4ffcb3b7cd884bd2ef8cdbbe29d6e228d189e99f1626f1"}
Jul 29 11:28:07.454830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.454781    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:07.471402 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:07.471349    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq" podStartSLOduration=1.471333452 podStartE2EDuration="1.471333452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:28:07.470039869 +0000 UTC m=+3563.650421751" watchObservedRunningTime="2026-07-29 11:28:07.471333452 +0000 UTC m=+3563.651715331"
Jul 29 11:28:38.461040 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:38.461009    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq"
Jul 29 11:28:46.299846 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:46.299749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:28:46.310756 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:28:46.310735    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:33:46.332620 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:33:46.332502    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:33:46.346818 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:33:46.346799    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:38:46.365958 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:38:46.365849    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:38:46.386572 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:38:46.386553    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:39:55.434517 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.434486    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:39:55.443791 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:39:55.443754    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:39:55.443791 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:55.443791 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:55.443791 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd" cmd=["/bin/sleep","15"]
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.443800    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" message="" err=<
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:39:55.443812    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" containerName="main" containerID="cri-o://b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"
Jul 29 11:39:55.444032 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.443854    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" containerName="main" containerID="cri-o://b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd" gracePeriod=60
Jul 29 11:39:55.701933 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.701909    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:39:55.761743 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761718    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.761873 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761757    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwj6f\" (UniqueName: \"kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.761873 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761776    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.761873 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761796    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.761873 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.761873 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761850    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir\") pod \"1d8221e3-fa2e-46cf-9760-c042190da938\" (UID: \"1d8221e3-fa2e-46cf-9760-c042190da938\") "
Jul 29 11:39:55.762128 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.761976    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.762128 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.762033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home" (OuterVolumeSpecName: "home") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.762128 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.762076    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.762310 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.762172    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.764113 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.764092    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm" (OuterVolumeSpecName: "dshm") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:39:55.764335 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.764309    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:39:55.764454 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.764341    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f" (OuterVolumeSpecName: "kube-api-access-zwj6f") pod "1d8221e3-fa2e-46cf-9760-c042190da938" (UID: "1d8221e3-fa2e-46cf-9760-c042190da938"). InnerVolumeSpecName "kube-api-access-zwj6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:39:55.862766 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.862741    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.862766 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.862760    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d8221e3-fa2e-46cf-9760-c042190da938-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.862766 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.862770    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.862938 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.862779    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwj6f\" (UniqueName: \"kubernetes.io/projected/1d8221e3-fa2e-46cf-9760-c042190da938-kube-api-access-zwj6f\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.862938 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.862787    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d8221e3-fa2e-46cf-9760-c042190da938-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:39:55.998131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.998059    2574 generic.go:356] "Generic (PLEG): container finished" podID="1d8221e3-fa2e-46cf-9760-c042190da938" containerID="b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd" exitCode=0
Jul 29 11:39:55.998131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.998100    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"1d8221e3-fa2e-46cf-9760-c042190da938","Type":"ContainerDied","Data":"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"}
Jul 29 11:39:55.998131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.998113    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"
Jul 29 11:39:55.998326 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.998171    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1" event={"ID":"1d8221e3-fa2e-46cf-9760-c042190da938","Type":"ContainerDied","Data":"d69eb13b83411df3a49c1f42f4316daa10c68d6dd31e05328c6ce1ec5968c730"}
Jul 29 11:39:55.998326 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:55.998192    2574 scope.go:117] "RemoveContainer" containerID="b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"
Jul 29 11:39:56.007962 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:56.007947    2574 scope.go:117] "RemoveContainer" containerID="b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"
Jul 29 11:39:56.008201 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:39:56.008185    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd\": container with ID starting with b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd not found: ID does not exist" containerID="b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"
Jul 29 11:39:56.008261 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:56.008207    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd"} err="failed to get container status \"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd\": rpc error: code = NotFound desc = could not find container \"b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd\": container with ID starting with b3e38ec56209deecd399f312b7d020cdcfbab2ba8ae06a8039d367b898975efd not found: ID does not exist"
Jul 29 11:39:56.019055 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:56.019031    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:39:56.020993 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:56.020975    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0-1"]
Jul 29 11:39:56.342707 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:39:56.342675    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" path="/var/lib/kubelet/pods/1d8221e3-fa2e-46cf-9760-c042190da938/volumes"
Jul 29 11:40:33.966358 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.966324    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:40:33.966787 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.966646    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" containerName="main"
Jul 29 11:40:33.966787 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.966656    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" containerName="main"
Jul 29 11:40:33.966787 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.966728    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8221e3-fa2e-46cf-9760-c042190da938" containerName="main"
Jul 29 11:40:33.969658 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.969641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:33.972440 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.972310    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 29 11:40:33.972440 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.972346    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 29 11:40:33.972440 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.972358    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-gp96d\""
Jul 29 11:40:33.972440 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.972420    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 29 11:40:33.972691 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.972454    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-j74zl\""
Jul 29 11:40:33.978416 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:33.978396    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:40:34.048270 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.048398 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.048398 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048383    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.048489 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.048489 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72mnp\" (UniqueName: \"kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.048595 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.048516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149479 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149655 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149655 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149655 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149834 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72mnp\" (UniqueName: \"kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149834 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149766    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149947 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.149947 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.149913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.150046 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.150022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.152634 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.152611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.152806 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.152790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.158620 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.158596    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72mnp\" (UniqueName: \"kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.280795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.280765    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:40:34.422364 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:34.422305    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:40:34.424595 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:40:34.424570    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc139953_3055_474b_8b62_2f5f559cd0c4.slice/crio-0388f093bc3e82281954ef52b8c2814b55a6a9741713c226f9a999e67f0d4a1c WatchSource:0}: Error finding container 0388f093bc3e82281954ef52b8c2814b55a6a9741713c226f9a999e67f0d4a1c: Status 404 returned error can't find the container with id 0388f093bc3e82281954ef52b8c2814b55a6a9741713c226f9a999e67f0d4a1c
Jul 29 11:40:35.201661 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:35.201578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"fc139953-3055-474b-8b62-2f5f559cd0c4","Type":"ContainerStarted","Data":"50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03"}
Jul 29 11:40:35.201661 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:35.201634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"fc139953-3055-474b-8b62-2f5f559cd0c4","Type":"ContainerStarted","Data":"0388f093bc3e82281954ef52b8c2814b55a6a9741713c226f9a999e67f0d4a1c"}
Jul 29 11:40:35.220534 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:40:35.220476    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=2.220456644 podStartE2EDuration="2.220456644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:40:35.218064914 +0000 UTC m=+4311.398446799" watchObservedRunningTime="2026-07-29 11:40:35.220456644 +0000 UTC m=+4311.400838526"
Jul 29 11:42:26.125632 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:42:26.125560    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7c3633d1_llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schesf9dq_e9a55162-d774-4af1-9b73-1b1121a3eb6c/main/0.log"
Jul 29 11:43:39.523547 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.523508    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:43:39.527085 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.527069    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.529196 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.529174    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 11:43:39.529331 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.529261    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 11:43:39.529385 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.529343    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cvv9k\""
Jul 29 11:43:39.529476 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.529455    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 11:43:39.529568 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.529555    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 29 11:43:39.545890 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.545861    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:43:39.635109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635252 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvqh\" (UniqueName: \"kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635252 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635252 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635372 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635372 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.635372 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.635306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.735860 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.735835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.735980 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.735867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.735980 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.735934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736076 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.735992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736076 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736076 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736248 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvqh\" (UniqueName: \"kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736309 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736309 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736296    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736418 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.736481 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.736463    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.738446 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.738425    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.738768 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.738750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.745325 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.745301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvqh\" (UniqueName: \"kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh\") pod \"stop-feature-test-kserve-d64ddc487-tr9l9\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.837009 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.836903    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:39.978759 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:39.978729    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:43:39.980429 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:43:39.980398    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafce3b8_5752_4217_98a6_4507fecc15cc.slice/crio-1904fdbcdf6a5487e943e3a3fd2b2fd6f38096f5be70b0f461a8c771f548cb25 WatchSource:0}: Error finding container 1904fdbcdf6a5487e943e3a3fd2b2fd6f38096f5be70b0f461a8c771f548cb25: Status 404 returned error can't find the container with id 1904fdbcdf6a5487e943e3a3fd2b2fd6f38096f5be70b0f461a8c771f548cb25
Jul 29 11:43:40.137335 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:40.137247    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerStarted","Data":"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"}
Jul 29 11:43:40.137335 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:40.137305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerStarted","Data":"1904fdbcdf6a5487e943e3a3fd2b2fd6f38096f5be70b0f461a8c771f548cb25"}
Jul 29 11:43:45.165319 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:45.165277    2574 generic.go:356] "Generic (PLEG): container finished" podID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerID="92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e" exitCode=0
Jul 29 11:43:45.165903 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:45.165356    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerDied","Data":"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"}
Jul 29 11:43:46.172168 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:46.172133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerStarted","Data":"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"}
Jul 29 11:43:46.195242 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:46.195171    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podStartSLOduration=2.855101843 podStartE2EDuration="7.195157019s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.340055176s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:43:46.191770349 +0000 UTC m=+4502.372152229" watchObservedRunningTime="2026-07-29 11:43:46.195157019 +0000 UTC m=+4502.375538898"
Jul 29 11:43:46.402119 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:46.402008    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:43:46.424507 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:46.424455    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:43:46.837856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:46.837824    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:49.837629 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:49.837590    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:43:49.839577 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:49.839543    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:43:59.837425 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:43:59.837381    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:09.837582 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:09.837544    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:19.838235 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:19.838171    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:29.837927 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:29.837892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:39.837978 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:39.837928    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:49.838097 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:49.838057    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:44:59.837653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:44:59.837611    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:45:09.838090 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:45:09.837995    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:45:19.837997 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:45:19.837952    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:45:29.837745 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:45:29.837701    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": dial tcp 10.132.0.65:8000: connect: connection refused"
Jul 29 11:45:39.847502 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:45:39.847467    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:45:39.855691 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:45:39.855666    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:46:06.008884 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:06.008850    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:46:21.031597 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.031539    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main" containerID="cri-o://9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945" gracePeriod=45
Jul 29 11:46:21.789505 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.789443    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:46:21.825404 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.825378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.825525 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.825413    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slvqh\" (UniqueName: \"kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.827749 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.827716    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:46:21.827749 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.827736    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh" (OuterVolumeSpecName: "kube-api-access-slvqh") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "kube-api-access-slvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:46:21.926336 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926311    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.926451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926384    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.926451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926402    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.926451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926422    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.926451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926443    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home\") pod \"eafce3b8-5752-4217-98a6-4507fecc15cc\" (UID: \"eafce3b8-5752-4217-98a6-4507fecc15cc\") "
Jul 29 11:46:21.926667 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926595    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:46:21.926716 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926679    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:21.926716 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926701    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eafce3b8-5752-4217-98a6-4507fecc15cc-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:21.926785 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.926713    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slvqh\" (UniqueName: \"kubernetes.io/projected/eafce3b8-5752-4217-98a6-4507fecc15cc-kube-api-access-slvqh\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:21.927109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.927087    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home" (OuterVolumeSpecName: "home") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:46:21.928445 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.928425    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm" (OuterVolumeSpecName: "dshm") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:46:21.937291 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.937268    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:46:21.987591 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:21.987554    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eafce3b8-5752-4217-98a6-4507fecc15cc" (UID: "eafce3b8-5752-4217-98a6-4507fecc15cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:46:22.004780 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.004752    2574 generic.go:356] "Generic (PLEG): container finished" podID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerID="9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945" exitCode=0
Jul 29 11:46:22.004900 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.004809    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"
Jul 29 11:46:22.004900 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.004807    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerDied","Data":"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"}
Jul 29 11:46:22.004900 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.004879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9" event={"ID":"eafce3b8-5752-4217-98a6-4507fecc15cc","Type":"ContainerDied","Data":"1904fdbcdf6a5487e943e3a3fd2b2fd6f38096f5be70b0f461a8c771f548cb25"}
Jul 29 11:46:22.005115 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.004916    2574 scope.go:117] "RemoveContainer" containerID="9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"
Jul 29 11:46:22.014781 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.014763    2574 scope.go:117] "RemoveContainer" containerID="92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"
Jul 29 11:46:22.023777 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.023753    2574 scope.go:117] "RemoveContainer" containerID="9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"
Jul 29 11:46:22.024069 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:46:22.024042    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945\": container with ID starting with 9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945 not found: ID does not exist" containerID="9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"
Jul 29 11:46:22.024151 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.024081    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945"} err="failed to get container status \"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945\": rpc error: code = NotFound desc = could not find container \"9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945\": container with ID starting with 9ad8d8e94df6d71ca14e402290b572389d1de9536f5594e459b4b82786f03945 not found: ID does not exist"
Jul 29 11:46:22.024151 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.024100    2574 scope.go:117] "RemoveContainer" containerID="92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"
Jul 29 11:46:22.024411 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:46:22.024394    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e\": container with ID starting with 92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e not found: ID does not exist" containerID="92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"
Jul 29 11:46:22.024517 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.024417    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e"} err="failed to get container status \"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e\": rpc error: code = NotFound desc = could not find container \"92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e\": container with ID starting with 92a78c5efeb0e5d6c8aca6396837a90f1666cf2287366cae69060367ef56c81e not found: ID does not exist"
Jul 29 11:46:22.026469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.026445    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:46:22.027556 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.027534    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:22.027650 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.027560    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:22.027650 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.027574    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:22.027650 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.027585    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eafce3b8-5752-4217-98a6-4507fecc15cc-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:46:22.029382 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.029363    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-tr9l9"]
Jul 29 11:46:22.342179 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.342151    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" path="/var/lib/kubelet/pods/eafce3b8-5752-4217-98a6-4507fecc15cc/volumes"
Jul 29 11:46:22.613079 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.612952    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:46:22.613469 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.613449    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="storage-initializer"
Jul 29 11:46:22.613529 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.613470    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="storage-initializer"
Jul 29 11:46:22.613529 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.613500    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main"
Jul 29 11:46:22.613529 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.613506    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main"
Jul 29 11:46:22.613643 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.613560    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eafce3b8-5752-4217-98a6-4507fecc15cc" containerName="main"
Jul 29 11:46:22.619953 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.619925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.623187 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.622988    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 29 11:46:22.623419 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.623391    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 29 11:46:22.623566 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.623548    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-cvv9k\""
Jul 29 11:46:22.623911 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.623890    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 29 11:46:22.624032 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.623946    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 29 11:46:22.626573 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.626555    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:46:22.632941 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.632918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.632956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdl8\" (UniqueName: \"kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.632981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.633004    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633258 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.633106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633258 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.633143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.633258 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.633237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.733933 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.733907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdl8\" (UniqueName: \"kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734058 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.733939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734058 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.733958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734058 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.733993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734058 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734271 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734271 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734408 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734470 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734435    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734470 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.734588 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.734554    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.736475 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.736451    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.736565 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.736534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.744317 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.744286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdl8\" (UniqueName: \"kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8\") pod \"stop-feature-test-kserve-d64ddc487-rn5ps\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:22.933899 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:22.933818    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:23.081443 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:23.081415    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:46:23.083434 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:46:23.083407    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119d7fc8_b726_4d12_a200_c039092038a2.slice/crio-929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83 WatchSource:0}: Error finding container 929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83: Status 404 returned error can't find the container with id 929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83
Jul 29 11:46:24.020971 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:24.020931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerStarted","Data":"b433234c216885b0c26cfe34af1160ed34f2a1b65d77b887dd3fd734f710e3df"}
Jul 29 11:46:24.021454 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:24.020987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerStarted","Data":"929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83"}
Jul 29 11:46:28.043914 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:28.043879    2574 generic.go:356] "Generic (PLEG): container finished" podID="119d7fc8-b726-4d12-a200-c039092038a2" containerID="b433234c216885b0c26cfe34af1160ed34f2a1b65d77b887dd3fd734f710e3df" exitCode=0
Jul 29 11:46:28.044402 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:28.043930    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerDied","Data":"b433234c216885b0c26cfe34af1160ed34f2a1b65d77b887dd3fd734f710e3df"}
Jul 29 11:46:29.052141 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:29.052094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerStarted","Data":"c54245d916aa36586c169caaa5a0cefd21c006bc13ad6d07f7bda9890742f750"}
Jul 29 11:46:29.073316 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:29.073265    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podStartSLOduration=2.7914185209999998 podStartE2EDuration="7.073211768s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.281793247s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:46:29.069304842 +0000 UTC m=+4665.249686722" watchObservedRunningTime="2026-07-29 11:46:29.073211768 +0000 UTC m=+4665.253593649"
Jul 29 11:46:29.934197 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:29.934165    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:32.934204 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:32.934165    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:46:32.935903 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:32.935872    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:46:42.934525 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:42.934441    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:46:52.934586 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:46:52.934541    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:02.935234 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:02.935185    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:12.934839 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:12.934796    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:22.934205 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:22.934166    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:32.935275 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:32.935234    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:42.935092 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:42.935053    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:47:52.934303 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:47:52.934254    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:48:02.934936 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:02.934892    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:48:12.934919 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:12.934822    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:48:22.947095 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:22.947066    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:48:22.955447 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:22.955423    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:48:46.435407 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:46.435304    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:48:46.459760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:48:46.459740    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:49:22.394576 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:22.394539    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:49:37.419592 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:37.419508    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" containerID="cri-o://c54245d916aa36586c169caaa5a0cefd21c006bc13ad6d07f7bda9890742f750" gracePeriod=45
Jul 29 11:49:37.948090 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:37.948058    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Jul 29 11:49:38.091809 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.091772    2574 generic.go:356] "Generic (PLEG): container finished" podID="119d7fc8-b726-4d12-a200-c039092038a2" containerID="c54245d916aa36586c169caaa5a0cefd21c006bc13ad6d07f7bda9890742f750" exitCode=0
Jul 29 11:49:38.091942 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.091810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerDied","Data":"c54245d916aa36586c169caaa5a0cefd21c006bc13ad6d07f7bda9890742f750"}
Jul 29 11:49:38.091942 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.091860    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps" event={"ID":"119d7fc8-b726-4d12-a200-c039092038a2","Type":"ContainerDied","Data":"929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83"}
Jul 29 11:49:38.091942 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.091880    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929d931e3050735eefd7152ab5a29a15d65b2b6e12713eff7be403ffe763bb83"
Jul 29 11:49:38.094322 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.094303    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:49:38.163718 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163652    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163718 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163679    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163718 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163701    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163722    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twdl8\" (UniqueName: \"kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163765    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163804    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs\") pod \"119d7fc8-b726-4d12-a200-c039092038a2\" (UID: \"119d7fc8-b726-4d12-a200-c039092038a2\") "
Jul 29 11:49:38.163949 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.163932    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:38.164291 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.164165    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.164456 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.164432    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home" (OuterVolumeSpecName: "home") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:38.166133 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.166111    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:49:38.166527 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.166508    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm" (OuterVolumeSpecName: "dshm") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:38.166600 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.166568    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8" (OuterVolumeSpecName: "kube-api-access-twdl8") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "kube-api-access-twdl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:49:38.179034 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.179009    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:38.231451 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.231419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "119d7fc8-b726-4d12-a200-c039092038a2" (UID: "119d7fc8-b726-4d12-a200-c039092038a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:49:38.265130 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265100    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.265130 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265128    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119d7fc8-b726-4d12-a200-c039092038a2-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.265317 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265142    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.265317 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265154    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.265317 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265165    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdl8\" (UniqueName: \"kubernetes.io/projected/119d7fc8-b726-4d12-a200-c039092038a2-kube-api-access-twdl8\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:38.265317 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:38.265176    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119d7fc8-b726-4d12-a200-c039092038a2-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:49:39.097075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:39.097047    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"
Jul 29 11:49:39.114143 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:39.114116    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:49:39.116698 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:39.116678    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-d64ddc487-rn5ps"]
Jul 29 11:49:40.342540 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:49:40.342508    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119d7fc8-b726-4d12-a200-c039092038a2" path="/var/lib/kubelet/pods/119d7fc8-b726-4d12-a200-c039092038a2/volumes"
Jul 29 11:50:07.641307 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641274    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:50:07.641760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641606    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main"
Jul 29 11:50:07.641760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641617    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main"
Jul 29 11:50:07.641760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641634    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="storage-initializer"
Jul 29 11:50:07.641760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641641    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="storage-initializer"
Jul 29 11:50:07.641760 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.641689    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="119d7fc8-b726-4d12-a200-c039092038a2" containerName="main"
Jul 29 11:50:07.644705 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.644684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.646435 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646404    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Jul 29 11:50:07.646658 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646640    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:50:07.646757 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646703    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-jfnls\""
Jul 29 11:50:07.646757 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646729    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Jul 29 11:50:07.646881 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646754    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 29 11:50:07.646881 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.646703    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Jul 29 11:50:07.654816 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.654796    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:50:07.684489 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684622 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684622 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684564    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684622 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684593    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56r5\" (UniqueName: \"kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.684795 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.684792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785492 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785492 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56r5\" (UniqueName: \"kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785713 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.785994 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.785739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.786052 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.786021    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.786137 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.786119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.786208 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.786166    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.786352 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.786335    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.786418 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.786373    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.787937 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.787911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.788561 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.788542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.793913 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.793887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56r5\" (UniqueName: \"kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5\") pod \"lora-single-adapter-test-kserve-5d5dd69664-f7q2f\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:07.956389 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:07.956324    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:08.100070 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:08.100010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:50:08.103894 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:50:08.103867    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b076a61_c923_4337_9552_81c49f317c03.slice/crio-271394e90bc3067921c8e6a5f94c5f74e54565e6df5d1ecd7b71c22b7790f380 WatchSource:0}: Error finding container 271394e90bc3067921c8e6a5f94c5f74e54565e6df5d1ecd7b71c22b7790f380: Status 404 returned error can't find the container with id 271394e90bc3067921c8e6a5f94c5f74e54565e6df5d1ecd7b71c22b7790f380
Jul 29 11:50:08.248392 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:08.248305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerStarted","Data":"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"}
Jul 29 11:50:08.248392 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:08.248369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerStarted","Data":"271394e90bc3067921c8e6a5f94c5f74e54565e6df5d1ecd7b71c22b7790f380"}
Jul 29 11:50:14.282756 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:14.282726    2574 generic.go:356] "Generic (PLEG): container finished" podID="1b076a61-c923-4337-9552-81c49f317c03" containerID="3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1" exitCode=0
Jul 29 11:50:14.283114 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:14.282774    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerDied","Data":"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"}
Jul 29 11:50:15.290920 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:15.290882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerStarted","Data":"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"}
Jul 29 11:50:15.312845 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:15.312791    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podStartSLOduration=2.808564902 podStartE2EDuration="8.312772958s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.504208056s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:50:15.310882315 +0000 UTC m=+4891.491264198" watchObservedRunningTime="2026-07-29 11:50:15.312772958 +0000 UTC m=+4891.493154838"
Jul 29 11:50:15.956651 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:15.956618    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:17.957062 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:17.957019    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:50:17.958844 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:17.958816    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:50:27.956801 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:27.956754    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:50:37.957179 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:37.957138    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:50:47.956925 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:47.956877    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:50:57.957302 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:50:57.957249    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:51:07.956984 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:51:07.956890    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:51:17.957674 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:51:17.957634    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 29 11:51:27.968725 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:51:27.968676    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:51:27.977776 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:51:27.977752    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:52:41.609238 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:52:41.609137    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5d5dd69664-f7q2f_1b076a61-c923-4337-9552-81c49f317c03/storage-initializer/0.log"
Jul 29 11:52:41.620758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:52:41.620721    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-5d5dd69664-f7q2f_1b076a61-c923-4337-9552-81c49f317c03/main/0.log"
Jul 29 11:52:44.982448 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:52:44.982411    2574 scope.go:117] "RemoveContainer" containerID="c54245d916aa36586c169caaa5a0cefd21c006bc13ad6d07f7bda9890742f750"
Jul 29 11:52:44.991640 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:52:44.991619    2574 scope.go:117] "RemoveContainer" containerID="b433234c216885b0c26cfe34af1160ed34f2a1b65d77b887dd3fd734f710e3df"
Jul 29 11:53:16.373646 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:16.373611    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:53:31.396501 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:31.396436    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" containerID="cri-o://b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3" gracePeriod=45
Jul 29 11:53:32.056615 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.056592    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:53:32.224374 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224288    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224374 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224346    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224374 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224381    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r56r5\" (UniqueName: \"kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224403    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224422    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224441    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224597    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224653 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm\") pod \"1b076a61-c923-4337-9552-81c49f317c03\" (UID: \"1b076a61-c923-4337-9552-81c49f317c03\") "
Jul 29 11:53:32.224914 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224820    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:53:32.224914 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224878    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home" (OuterVolumeSpecName: "home") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:32.225021 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224971    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b076a61-c923-4337-9552-81c49f317c03-cabundle-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.225021 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.224986    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.225021 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.225012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:32.227156 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.227131    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm" (OuterVolumeSpecName: "dshm") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:32.227295 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.227140    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:53:32.227295 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.227233    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5" (OuterVolumeSpecName: "kube-api-access-r56r5") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "kube-api-access-r56r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:53:32.235566 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.235544    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:32.288908 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.288874    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b076a61-c923-4337-9552-81c49f317c03" (UID: "1b076a61-c923-4337-9552-81c49f317c03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:53:32.325625 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325603    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.325625 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325623    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.325758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325633    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56r5\" (UniqueName: \"kubernetes.io/projected/1b076a61-c923-4337-9552-81c49f317c03-kube-api-access-r56r5\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.325758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325641    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.325758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325651    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b076a61-c923-4337-9552-81c49f317c03-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.325758 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.325659    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b076a61-c923-4337-9552-81c49f317c03-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:53:32.404479 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.404449    2574 generic.go:356] "Generic (PLEG): container finished" podID="1b076a61-c923-4337-9552-81c49f317c03" containerID="b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3" exitCode=0
Jul 29 11:53:32.404856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.404504    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"
Jul 29 11:53:32.404856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.404517    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerDied","Data":"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"}
Jul 29 11:53:32.404856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.404581    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" event={"ID":"1b076a61-c923-4337-9552-81c49f317c03","Type":"ContainerDied","Data":"271394e90bc3067921c8e6a5f94c5f74e54565e6df5d1ecd7b71c22b7790f380"}
Jul 29 11:53:32.404856 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.404618    2574 scope.go:117] "RemoveContainer" containerID="b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"
Jul 29 11:53:32.416274 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.416243    2574 scope.go:117] "RemoveContainer" containerID="3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"
Jul 29 11:53:32.421116 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.421098    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:53:32.425595 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.425573    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f"]
Jul 29 11:53:32.427074 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.427057    2574 scope.go:117] "RemoveContainer" containerID="b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"
Jul 29 11:53:32.427383 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:53:32.427365    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3\": container with ID starting with b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3 not found: ID does not exist" containerID="b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"
Jul 29 11:53:32.427436 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.427391    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3"} err="failed to get container status \"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3\": rpc error: code = NotFound desc = could not find container \"b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3\": container with ID starting with b61723979f7134d35cfb6521cb21d899f495ec11e736fa4bbbb64a72d126b0f3 not found: ID does not exist"
Jul 29 11:53:32.427436 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.427409    2574 scope.go:117] "RemoveContainer" containerID="3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"
Jul 29 11:53:32.427637 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:53:32.427617    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1\": container with ID starting with 3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1 not found: ID does not exist" containerID="3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"
Jul 29 11:53:32.427682 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.427644    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1"} err="failed to get container status \"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1\": rpc error: code = NotFound desc = could not find container \"3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1\": container with ID starting with 3f622157febfe352b292689a550329adf4a58c44cfeb97deccc51464d6a68af1 not found: ID does not exist"
Jul 29 11:53:32.970595 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:32.970525    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-5d5dd69664-f7q2f" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Jul 29 11:53:34.343407 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:34.343373    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b076a61-c923-4337-9552-81c49f317c03" path="/var/lib/kubelet/pods/1b076a61-c923-4337-9552-81c49f317c03/volumes"
Jul 29 11:53:45.540750 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.540711    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:53:45.541261 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.541239    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="storage-initializer"
Jul 29 11:53:45.541343 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.541263    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="storage-initializer"
Jul 29 11:53:45.541343 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.541297    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main"
Jul 29 11:53:45.541343 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.541306    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main"
Jul 29 11:53:45.541512 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.541402    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b076a61-c923-4337-9552-81c49f317c03" containerName="main"
Jul 29 11:53:45.545705 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.545684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.547457 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547431    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-mcw6h\""
Jul 29 11:53:45.547588 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547516    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Jul 29 11:53:45.547588 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547536    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Jul 29 11:53:45.547690 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547672    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Jul 29 11:53:45.547893 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547875    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Jul 29 11:53:45.547933 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.547888    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 29 11:53:45.555358 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.555337    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:53:45.725109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725109 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rtqm\" (UniqueName: \"kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725297    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725347 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.725511 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.725353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826631 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826537    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826631 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826586    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826810 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rtqm\" (UniqueName: \"kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826810 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826850    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.826916 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827074 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827074 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.826997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827261 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.827241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827368 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.827302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827431 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.827379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.827787 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.827766    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.829099 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.829080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.829319 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.829302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.837055 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.837036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rtqm\" (UniqueName: \"kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm\") pod \"lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.856874 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.856849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:45.999876 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:45.999794    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:53:46.002102 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:53:46.002076    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d958dd3_4c68_45ac_9881_c893685badf0.slice/crio-ae625be9c7129abb38bdc78f731c99cd65be24d6666d7a5e43996e44808c85e0 WatchSource:0}: Error finding container ae625be9c7129abb38bdc78f731c99cd65be24d6666d7a5e43996e44808c85e0: Status 404 returned error can't find the container with id ae625be9c7129abb38bdc78f731c99cd65be24d6666d7a5e43996e44808c85e0
Jul 29 11:53:46.478619 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:46.478500    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:53:46.493382 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:46.481452    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerStarted","Data":"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"}
Jul 29 11:53:46.493382 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:46.481523    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerStarted","Data":"ae625be9c7129abb38bdc78f731c99cd65be24d6666d7a5e43996e44808c85e0"}
Jul 29 11:53:46.503509 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:46.503482    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:53:51.510399 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:51.510367    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d958dd3-4c68-45ac-9881-c893685badf0" containerID="8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237" exitCode=0
Jul 29 11:53:51.510757 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:51.510424    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerDied","Data":"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"}
Jul 29 11:53:52.518497 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:52.518461    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerStarted","Data":"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"}
Jul 29 11:53:52.539487 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:52.539417    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podStartSLOduration=2.596231166 podStartE2EDuration="7.539397028s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.943165862s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:53:52.537671786 +0000 UTC m=+5108.718053667" watchObservedRunningTime="2026-07-29 11:53:52.539397028 +0000 UTC m=+5108.719778910"
Jul 29 11:53:52.857343 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:52.857265    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:55.857273 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:55.857244    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:53:55.859052 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:53:55.859025    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:05.857704 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:05.857617    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:15.857820 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:15.857784    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:25.857932 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:25.857891    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:35.857784 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:35.857745    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:45.857185 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:45.857147    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:54:55.857465 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:54:55.857417    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:55:05.867997 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:05.867967    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:55:05.876099 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:05.876075    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:55:24.802690 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:24.802656    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:55:24.811807 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:55:24.811767    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 11:55:24.811807 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.811807 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.811807 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03" cmd=["/bin/sleep","15"]
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:24.811815    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:55:24.811827    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerName="main" containerID="cri-o://50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03"
Jul 29 11:55:24.812332 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:24.811862    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerName="main" containerID="cri-o://50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03" gracePeriod=60
Jul 29 11:55:25.042994 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.042961    2574 generic.go:356] "Generic (PLEG): container finished" podID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerID="50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03" exitCode=0
Jul 29 11:55:25.043157 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.043051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"fc139953-3055-474b-8b62-2f5f559cd0c4","Type":"ContainerDied","Data":"50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03"}
Jul 29 11:55:25.070812 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.070785    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:55:25.200911 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.200874    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.200927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.200952    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72mnp\" (UniqueName: \"kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.200970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.200994    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201075 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201065    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs\") pod \"fc139953-3055-474b-8b62-2f5f559cd0c4\" (UID: \"fc139953-3055-474b-8b62-2f5f559cd0c4\") "
Jul 29 11:55:25.201379 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201186    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:25.201379 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201212    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:25.201379 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201286    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home" (OuterVolumeSpecName: "home") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:25.201557 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201549    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:25.201614 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201565    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:25.201614 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.201578    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:25.203430 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.203410    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm" (OuterVolumeSpecName: "dshm") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:55:25.203797 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.203780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp" (OuterVolumeSpecName: "kube-api-access-72mnp") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "kube-api-access-72mnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:55:25.203874 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.203787    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc139953-3055-474b-8b62-2f5f559cd0c4" (UID: "fc139953-3055-474b-8b62-2f5f559cd0c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:55:25.302553 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.302527    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc139953-3055-474b-8b62-2f5f559cd0c4-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:25.302553 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.302552    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72mnp\" (UniqueName: \"kubernetes.io/projected/fc139953-3055-474b-8b62-2f5f559cd0c4-kube-api-access-72mnp\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:25.302733 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:25.302563    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc139953-3055-474b-8b62-2f5f559cd0c4-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:55:26.050531 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.050504    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 29 11:55:26.050989 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.050539    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"fc139953-3055-474b-8b62-2f5f559cd0c4","Type":"ContainerDied","Data":"0388f093bc3e82281954ef52b8c2814b55a6a9741713c226f9a999e67f0d4a1c"}
Jul 29 11:55:26.050989 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.050609    2574 scope.go:117] "RemoveContainer" containerID="50f9bc264175f3aced95ee4ea1372430fc66bd88b9344c31be7102f7a7b20c03"
Jul 29 11:55:26.070894 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.070872    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:55:26.074924 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.074902    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 29 11:55:26.342568 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:26.342484    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" path="/var/lib/kubelet/pods/fc139953-3055-474b-8b62-2f5f559cd0c4/volumes"
Jul 29 11:55:56.787367 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:56.787337    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6_4d958dd3-4c68-45ac-9881-c893685badf0/storage-initializer/0.log"
Jul 29 11:55:56.794869 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:55:56.794843    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6_4d958dd3-4c68-45ac-9881-c893685badf0/main/0.log"
Jul 29 11:56:16.450416 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:16.450378    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:56:31.475717 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:31.475654    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" containerID="cri-o://fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004" gracePeriod=45
Jul 29 11:56:31.868615 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:31.868569    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Jul 29 11:56:32.139641 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.139611    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:56:32.272602 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272574    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272743 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272627    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rtqm\" (UniqueName: \"kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272743 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272668    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272743 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272683    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272743 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272712    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272954 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272752    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.272954 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272797    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.273041 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.272999    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm\") pod \"4d958dd3-4c68-45ac-9881-c893685badf0\" (UID: \"4d958dd3-4c68-45ac-9881-c893685badf0\") "
Jul 29 11:56:32.273091 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273071    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:32.273131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273082    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 11:56:32.273131 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273121    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home" (OuterVolumeSpecName: "home") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:32.273386 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273368    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.273442 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273393    2574 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d958dd3-4c68-45ac-9881-c893685badf0-cabundle-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.273442 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.273406    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.275687 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.275597    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm" (OuterVolumeSpecName: "kube-api-access-5rtqm") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "kube-api-access-5rtqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:56:32.275830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.275712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:56:32.275830 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.275719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm" (OuterVolumeSpecName: "dshm") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:32.283348 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.283324    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:32.338386 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.338355    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d958dd3-4c68-45ac-9881-c893685badf0" (UID: "4d958dd3-4c68-45ac-9881-c893685badf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:56:32.374018 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.373958    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.374018 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.373980    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rtqm\" (UniqueName: \"kubernetes.io/projected/4d958dd3-4c68-45ac-9881-c893685badf0-kube-api-access-5rtqm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.374018 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.373995    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.374018 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.374008    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d958dd3-4c68-45ac-9881-c893685badf0-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.374018 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.374020    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d958dd3-4c68-45ac-9881-c893685badf0-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:56:32.416904 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.416876    2574 generic.go:356] "Generic (PLEG): container finished" podID="4d958dd3-4c68-45ac-9881-c893685badf0" containerID="fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004" exitCode=0
Jul 29 11:56:32.417005 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.416917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerDied","Data":"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"}
Jul 29 11:56:32.417005 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.416943    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"
Jul 29 11:56:32.417005 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.416962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6" event={"ID":"4d958dd3-4c68-45ac-9881-c893685badf0","Type":"ContainerDied","Data":"ae625be9c7129abb38bdc78f731c99cd65be24d6666d7a5e43996e44808c85e0"}
Jul 29 11:56:32.417005 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.416984    2574 scope.go:117] "RemoveContainer" containerID="fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"
Jul 29 11:56:32.425667 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.425649    2574 scope.go:117] "RemoveContainer" containerID="8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"
Jul 29 11:56:32.434797 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.434774    2574 scope.go:117] "RemoveContainer" containerID="fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"
Jul 29 11:56:32.435075 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:56:32.435052    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004\": container with ID starting with fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004 not found: ID does not exist" containerID="fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"
Jul 29 11:56:32.435176 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.435086    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004"} err="failed to get container status \"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004\": rpc error: code = NotFound desc = could not find container \"fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004\": container with ID starting with fd9d94904f7e56d6c6c6f1602b27b4fbf1948d6c055df667b4b22106c9c2a004 not found: ID does not exist"
Jul 29 11:56:32.435176 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.435108    2574 scope.go:117] "RemoveContainer" containerID="8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"
Jul 29 11:56:32.435434 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:56:32.435406    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237\": container with ID starting with 8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237 not found: ID does not exist" containerID="8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"
Jul 29 11:56:32.435539 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.435438    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237"} err="failed to get container status \"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237\": rpc error: code = NotFound desc = could not find container \"8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237\": container with ID starting with 8758a6e5845d2edaa8aff37cb371c6afd8fd9380731bb51b4b17ccbc76733237 not found: ID does not exist"
Jul 29 11:56:32.436545 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.436523    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:56:32.442345 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:32.442325    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-55f8fc498f-zw6b6"]
Jul 29 11:56:34.342834 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:56:34.342800    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" path="/var/lib/kubelet/pods/4d958dd3-4c68-45ac-9881-c893685badf0/volumes"
Jul 29 11:57:17.315278 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315186    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315631    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerName="main"
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315651    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerName="main"
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315676    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="storage-initializer"
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315681    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="storage-initializer"
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315691    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main"
Jul 29 11:57:17.315748 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main"
Jul 29 11:57:17.316031 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315757    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d958dd3-4c68-45ac-9881-c893685badf0" containerName="main"
Jul 29 11:57:17.316031 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.315770    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc139953-3055-474b-8b62-2f5f559cd0c4" containerName="main"
Jul 29 11:57:17.318592 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.318572    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.320922 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.320899    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Jul 29 11:57:17.321231 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.321196    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-x6gvf\""
Jul 29 11:57:17.321364 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.321344    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 29 11:57:17.321429 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.321350    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 29 11:57:17.321500 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.321485    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 29 11:57:17.331789 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.331756    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:57:17.414756 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414728    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.414861 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.414906 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.414947 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.414947 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.415014 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.415014 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.414967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.515969 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.515941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516087 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.515992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516087 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516087 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516280 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516280 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516280 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516460 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516521 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516490    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516580 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.516715 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.516698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.519069 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.519040    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.519242 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.519209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.530591 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.530572    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89\") pod \"tls-verification-test-kserve-6b95d7b7d-bcksb\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.628796 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.628723    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:17.769797 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:17.769766    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:57:17.773586 ip-10-0-133-86 kubenswrapper[2574]: W0729 11:57:17.773559    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a7c307_c385_4621_b137_4e9d1323b78e.slice/crio-f23f64469496db3a41034b228d57daed2ac31adcbc778f0f44b6ec1aad2a7f09 WatchSource:0}: Error finding container f23f64469496db3a41034b228d57daed2ac31adcbc778f0f44b6ec1aad2a7f09: Status 404 returned error can't find the container with id f23f64469496db3a41034b228d57daed2ac31adcbc778f0f44b6ec1aad2a7f09
Jul 29 11:57:18.657902 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:18.657870    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerStarted","Data":"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"}
Jul 29 11:57:18.658304 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:18.657919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerStarted","Data":"f23f64469496db3a41034b228d57daed2ac31adcbc778f0f44b6ec1aad2a7f09"}
Jul 29 11:57:22.680113 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:22.680075    2574 generic.go:356] "Generic (PLEG): container finished" podID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerID="1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e" exitCode=0
Jul 29 11:57:22.680520 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:22.680146    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerDied","Data":"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"}
Jul 29 11:57:23.690925 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:23.690888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerStarted","Data":"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"}
Jul 29 11:57:23.725749 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:23.725695    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podStartSLOduration=2.533661609 podStartE2EDuration="6.725680088s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.192018479s" isStatefulPod=false podCreationTimestamp="2026-07-29 11:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 11:57:23.724944457 +0000 UTC m=+5319.905326338" watchObservedRunningTime="2026-07-29 11:57:23.725680088 +0000 UTC m=+5319.906062002"
Jul 29 11:57:24.629309 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:24.629276    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:27.629757 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:27.629724    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:57:27.631085 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:27.631057    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:57:37.630015 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:37.629962    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:57:47.630112 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:47.630066    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:57:57.630156 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:57:57.630115    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:07.629440 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:07.629396    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:17.629595 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:17.629553    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:27.629530 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:27.629492    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:37.629571 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:37.629521    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:46.518524 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:46.518497    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:58:46.551159 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:46.551133    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 11:58:47.630002 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:47.629962    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:58:57.629176 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:58:57.629131    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:59:07.629290 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:07.629253    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:59:17.639525 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:17.639494    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:59:17.647290 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:17.647269    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:59:35.285327 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:35.285289    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:59:50.308581 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:50.308542    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" containerID="cri-o://172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c" gracePeriod=45
Jul 29 11:59:50.640903 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:50.640814    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Jul 29 11:59:51.072799 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.072773    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:59:51.094355 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094327    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094387    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094413    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094442    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094484 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094480    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094770 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm\") pod \"c1a7c307-c385-4621-b137-4e9d1323b78e\" (UID: \"c1a7c307-c385-4621-b137-4e9d1323b78e\") "
Jul 29 11:59:51.094770 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094719    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:51.094939 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.094776    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.095805 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.095766    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home" (OuterVolumeSpecName: "home") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:51.097732 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.097703    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89" (OuterVolumeSpecName: "kube-api-access-54k89") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "kube-api-access-54k89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 11:59:51.098074 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.098053    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm" (OuterVolumeSpecName: "dshm") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:51.098162 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.098116    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 11:59:51.111410 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.111386    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:51.163468 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.163433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1a7c307-c385-4621-b137-4e9d1323b78e" (UID: "c1a7c307-c385-4621-b137-4e9d1323b78e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195454    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54k89\" (UniqueName: \"kubernetes.io/projected/c1a7c307-c385-4621-b137-4e9d1323b78e-kube-api-access-54k89\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195478    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195487    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195497    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195505    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1a7c307-c385-4621-b137-4e9d1323b78e-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.195516 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.195514    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1a7c307-c385-4621-b137-4e9d1323b78e-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 11:59:51.490853 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.490780    2574 generic.go:356] "Generic (PLEG): container finished" podID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerID="172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c" exitCode=0
Jul 29 11:59:51.490853 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.490831    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"
Jul 29 11:59:51.491290 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.490825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerDied","Data":"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"}
Jul 29 11:59:51.491290 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.490948    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb" event={"ID":"c1a7c307-c385-4621-b137-4e9d1323b78e","Type":"ContainerDied","Data":"f23f64469496db3a41034b228d57daed2ac31adcbc778f0f44b6ec1aad2a7f09"}
Jul 29 11:59:51.491290 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.490973    2574 scope.go:117] "RemoveContainer" containerID="172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"
Jul 29 11:59:51.504201 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.504184    2574 scope.go:117] "RemoveContainer" containerID="1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"
Jul 29 11:59:51.512362 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.512346    2574 scope.go:117] "RemoveContainer" containerID="172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"
Jul 29 11:59:51.512778 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:59:51.512747    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c\": container with ID starting with 172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c not found: ID does not exist" containerID="172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"
Jul 29 11:59:51.512867 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.512775    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c"} err="failed to get container status \"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c\": rpc error: code = NotFound desc = could not find container \"172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c\": container with ID starting with 172f3e67e9ad0ee92b378fd2aef8707197a5351baf1d308b16a6a705b8f8d63c not found: ID does not exist"
Jul 29 11:59:51.512867 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.512816    2574 scope.go:117] "RemoveContainer" containerID="1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"
Jul 29 11:59:51.513115 ip-10-0-133-86 kubenswrapper[2574]: E0729 11:59:51.513059    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e\": container with ID starting with 1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e not found: ID does not exist" containerID="1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"
Jul 29 11:59:51.513115 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.513099    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e"} err="failed to get container status \"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e\": rpc error: code = NotFound desc = could not find container \"1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e\": container with ID starting with 1bf2bdaaa25d828ff51713e8bfbaa1feb30bce985153aa4f260fb38166ad575e not found: ID does not exist"
Jul 29 11:59:51.515744 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.515723    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:59:51.518904 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:51.518885    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-6b95d7b7d-bcksb"]
Jul 29 11:59:52.342094 ip-10-0-133-86 kubenswrapper[2574]: I0729 11:59:52.342066    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" path="/var/lib/kubelet/pods/c1a7c307-c385-4621-b137-4e9d1323b78e/volumes"
Jul 29 12:00:31.933023 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.932991    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:00:31.933542 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.933445    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="storage-initializer"
Jul 29 12:00:31.933542 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.933465    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="storage-initializer"
Jul 29 12:00:31.933542 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.933489    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main"
Jul 29 12:00:31.933542 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.933495    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main"
Jul 29 12:00:31.933683 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.933567    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a7c307-c385-4621-b137-4e9d1323b78e" containerName="main"
Jul 29 12:00:31.936701 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.936682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.938837 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.938652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tntgg\""
Jul 29 12:00:31.938837 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.938720    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 29 12:00:31.938837 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.938720    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 29 12:00:31.938837 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.938793    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 29 12:00:31.939141 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.938907    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 29 12:00:31.947208 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.947188    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:00:31.989638 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989596    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989763 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbbl\" (UniqueName: \"kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989763 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989763 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989905 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989905 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:31.989905 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:31.989889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091068 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbbl\" (UniqueName: \"kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091189 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091189 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091189 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091113    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091189 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091145    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091189 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091496 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091554 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091524    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091662 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091643    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091718 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.091718 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.091688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.093717 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.093689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.093941 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.093925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.098101 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.098076    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbbl\" (UniqueName: \"kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl\") pod \"prestop-hook-test-kserve-c98c474bb-mwxm8\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.247887 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.247813    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:32.391292 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.391260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:00:32.394771 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:00:32.394741    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51b094e_cecc_43ea_a352_0b38404b4590.slice/crio-572fc1aae5cf5cc0fcfcb7153d14011bb4ba77d27aef5f9f036c4683e30a8cea WatchSource:0}: Error finding container 572fc1aae5cf5cc0fcfcb7153d14011bb4ba77d27aef5f9f036c4683e30a8cea: Status 404 returned error can't find the container with id 572fc1aae5cf5cc0fcfcb7153d14011bb4ba77d27aef5f9f036c4683e30a8cea
Jul 29 12:00:32.707338 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.707308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerStarted","Data":"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"}
Jul 29 12:00:32.707533 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:32.707355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerStarted","Data":"572fc1aae5cf5cc0fcfcb7153d14011bb4ba77d27aef5f9f036c4683e30a8cea"}
Jul 29 12:00:36.733884 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:36.733806    2574 generic.go:356] "Generic (PLEG): container finished" podID="c51b094e-cecc-43ea-a352-0b38404b4590" containerID="d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375" exitCode=0
Jul 29 12:00:36.733884 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:36.733843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerDied","Data":"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"}
Jul 29 12:00:37.745701 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:37.745662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerStarted","Data":"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"}
Jul 29 12:00:37.767791 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:37.767731    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podStartSLOduration=2.764197194 podStartE2EDuration="6.767712474s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00351528s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:00:37.765028608 +0000 UTC m=+5513.945410495" watchObservedRunningTime="2026-07-29 12:00:37.767712474 +0000 UTC m=+5513.948094355"
Jul 29 12:00:38.248029 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:38.247991    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:42.248704 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:42.248665    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:00:42.250079 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:42.250050    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:00:52.249163 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:00:52.249117    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:02.249090 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:02.249041    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:12.249058 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:12.249013    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:22.249074 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:22.249029    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:32.249169 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:32.249128    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:42.248884 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:42.248800    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:01:52.249024 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:01:52.248983    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:02:02.248802 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:02.248764    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:02:12.248966 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:12.248919    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:02:22.248971 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:22.248923    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:02:32.258428 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:32.258398    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:02:32.266545 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:32.266519    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:02:33.860846 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.860806    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:02:33.885532 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.885496    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:02:33.890454 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.890435    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.899832 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.899806    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:02:33.914001 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.913979    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914135 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914135 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914097    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914281 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914135    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914281 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914281 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:33.914414 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:33.914293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7787s\" (UniqueName: \"kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014603 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014713 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014713 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7787s\" (UniqueName: \"kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014836 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014836 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014951 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.014951 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.015068 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.014963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.015068 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.015026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.015068 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.015054    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.015204 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.015184    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.017137 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.017110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.017375 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.017359    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.021988 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.021969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7787s\" (UniqueName: \"kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s\") pod \"prestop-hook-test-kserve-c98c474bb-tdxdt\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.202388 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.202310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:34.347284 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.347260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:02:34.348716 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:02:34.348690    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcf1b3f_7f9e_48f6_85f1_b3277115352f.slice/crio-e4bb2db29a65f530136abbae29d3da22c690d34fda352f64fef399f015557c5f WatchSource:0}: Error finding container e4bb2db29a65f530136abbae29d3da22c690d34fda352f64fef399f015557c5f: Status 404 returned error can't find the container with id e4bb2db29a65f530136abbae29d3da22c690d34fda352f64fef399f015557c5f
Jul 29 12:02:34.372750 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:34.372723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerStarted","Data":"e4bb2db29a65f530136abbae29d3da22c690d34fda352f64fef399f015557c5f"}
Jul 29 12:02:35.380832 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:35.380795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerStarted","Data":"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"}
Jul 29 12:02:39.405968 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:39.405936    2574 generic.go:356] "Generic (PLEG): container finished" podID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerID="8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce" exitCode=0
Jul 29 12:02:39.406367 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:39.406011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerDied","Data":"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"}
Jul 29 12:02:40.413209 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:40.413178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerStarted","Data":"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"}
Jul 29 12:02:40.432691 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:40.432643    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podStartSLOduration=3.019963695 podStartE2EDuration="7.432627994s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.412664299s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:02:40.43157084 +0000 UTC m=+5636.611952719" watchObservedRunningTime="2026-07-29 12:02:40.432627994 +0000 UTC m=+5636.613009873"
Jul 29 12:02:41.203212 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:41.203179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:44.203471 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:44.203429    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:02:44.204833 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:44.204806    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 29 12:02:48.883210 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:48.883165    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" containerID="cri-o://27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77" gracePeriod=45
Jul 29 12:02:49.259033 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.258928    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Jul 29 12:02:49.681801 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.681767    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:02:49.748516 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748537    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748571    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748607    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzbbl\" (UniqueName: \"kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748645    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748673    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748992 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748738    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir\") pod \"c51b094e-cecc-43ea-a352-0b38404b4590\" (UID: \"c51b094e-cecc-43ea-a352-0b38404b4590\") "
Jul 29 12:02:49.748992 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748873    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache" (OuterVolumeSpecName: "model-cache") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:02:49.748992 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.748974    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.749279 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.749209    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home" (OuterVolumeSpecName: "home") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:02:49.751568 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.751527    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl" (OuterVolumeSpecName: "kube-api-access-bzbbl") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "kube-api-access-bzbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:02:49.751568 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.751551    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:02:49.751815 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.751791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm" (OuterVolumeSpecName: "dshm") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:02:49.761658 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.761635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:02:49.808307 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.808278    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c51b094e-cecc-43ea-a352-0b38404b4590" (UID: "c51b094e-cecc-43ea-a352-0b38404b4590"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:02:49.849486 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849453    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.849486 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849484    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.849486 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849495    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.849671 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849503    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzbbl\" (UniqueName: \"kubernetes.io/projected/c51b094e-cecc-43ea-a352-0b38404b4590-kube-api-access-bzbbl\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.849671 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849511    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c51b094e-cecc-43ea-a352-0b38404b4590-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:49.849671 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:49.849530    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c51b094e-cecc-43ea-a352-0b38404b4590-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:02:50.474151 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.474115    2574 generic.go:356] "Generic (PLEG): container finished" podID="c51b094e-cecc-43ea-a352-0b38404b4590" containerID="27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77" exitCode=0
Jul 29 12:02:50.474589 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.474171    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"
Jul 29 12:02:50.474589 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.474170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerDied","Data":"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"}
Jul 29 12:02:50.474589 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.474251    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8" event={"ID":"c51b094e-cecc-43ea-a352-0b38404b4590","Type":"ContainerDied","Data":"572fc1aae5cf5cc0fcfcb7153d14011bb4ba77d27aef5f9f036c4683e30a8cea"}
Jul 29 12:02:50.474589 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.474287    2574 scope.go:117] "RemoveContainer" containerID="27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"
Jul 29 12:02:50.483678 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.483659    2574 scope.go:117] "RemoveContainer" containerID="d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"
Jul 29 12:02:50.491145 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.491123    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:02:50.494500 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.494477    2574 scope.go:117] "RemoveContainer" containerID="27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"
Jul 29 12:02:50.494736 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.494718    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-mwxm8"]
Jul 29 12:02:50.494803 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:02:50.494789    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77\": container with ID starting with 27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77 not found: ID does not exist" containerID="27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"
Jul 29 12:02:50.494848 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.494815    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77"} err="failed to get container status \"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77\": rpc error: code = NotFound desc = could not find container \"27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77\": container with ID starting with 27d69a348c6443b8f625539d4800f629c41f3161ab8082e48e43beba57a97c77 not found: ID does not exist"
Jul 29 12:02:50.494848 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.494839    2574 scope.go:117] "RemoveContainer" containerID="d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"
Jul 29 12:02:50.495127 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:02:50.495111    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375\": container with ID starting with d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375 not found: ID does not exist" containerID="d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"
Jul 29 12:02:50.495187 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:50.495131    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375"} err="failed to get container status \"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375\": rpc error: code = NotFound desc = could not find container \"d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375\": container with ID starting with d6feb848ad4e7a1a972a028e638de818e8a4abea7ac45e11854689841701c375 not found: ID does not exist"
Jul 29 12:02:52.342862 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:52.342825    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" path="/var/lib/kubelet/pods/c51b094e-cecc-43ea-a352-0b38404b4590/volumes"
Jul 29 12:02:54.203581 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:02:54.203528    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 29 12:03:04.203252 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:04.203195    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 29 12:03:14.203239 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:14.203117    2574 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 29 12:03:22.023773 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:22.023737    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:03:37.047525 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:37.047458    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main" containerID="cri-o://ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe" gracePeriod=45
Jul 29 12:03:46.554915 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:46.554800    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 12:03:46.587894 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:03:46.587860    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 12:04:22.706898 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.706875    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-c98c474bb-tdxdt_1dcf1b3f-7f9e-48f6-85f1-b3277115352f/main/0.log"
Jul 29 12:04:22.709648 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.709630    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:04:22.830177 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830103    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830177 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830139    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830177 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830163    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830464 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830197    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7787s\" (UniqueName: \"kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830464 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830363    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830464 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830426    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830633 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm\") pod \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\" (UID: \"1dcf1b3f-7f9e-48f6-85f1-b3277115352f\") "
Jul 29 12:04:22.830633 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830544    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:04:22.830633 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830567    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home" (OuterVolumeSpecName: "home") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:04:22.830633 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830584    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:04:22.830857 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830744    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.830857 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830764    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.830857 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.830777    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.832530 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.832501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s" (OuterVolumeSpecName: "kube-api-access-7787s") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "kube-api-access-7787s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:04:22.832632 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.832547    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:04:22.832777 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.832757    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm" (OuterVolumeSpecName: "dshm") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:04:22.890960 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.890929    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1dcf1b3f-7f9e-48f6-85f1-b3277115352f" (UID: "1dcf1b3f-7f9e-48f6-85f1-b3277115352f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:04:22.931175 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.931148    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7787s\" (UniqueName: \"kubernetes.io/projected/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kube-api-access-7787s\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.931175 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.931171    2574 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-kserve-provision-location\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.931370 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.931182    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.931370 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.931191    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dcf1b3f-7f9e-48f6-85f1-b3277115352f-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:04:22.979355 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979338    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-c98c474bb-tdxdt_1dcf1b3f-7f9e-48f6-85f1-b3277115352f/main/0.log"
Jul 29 12:04:22.979707 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979688    2574 generic.go:356] "Generic (PLEG): container finished" podID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerID="ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe" exitCode=137
Jul 29 12:04:22.979776 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979743    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"
Jul 29 12:04:22.979776 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979756    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerDied","Data":"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"}
Jul 29 12:04:22.979908 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979828    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt" event={"ID":"1dcf1b3f-7f9e-48f6-85f1-b3277115352f","Type":"ContainerDied","Data":"e4bb2db29a65f530136abbae29d3da22c690d34fda352f64fef399f015557c5f"}
Jul 29 12:04:22.979908 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.979852    2574 scope.go:117] "RemoveContainer" containerID="ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"
Jul 29 12:04:22.989103 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.989081    2574 scope.go:117] "RemoveContainer" containerID="8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"
Jul 29 12:04:22.997386 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.997369    2574 scope.go:117] "RemoveContainer" containerID="ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"
Jul 29 12:04:22.997637 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:04:22.997616    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe\": container with ID starting with ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe not found: ID does not exist" containerID="ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"
Jul 29 12:04:22.997686 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.997645    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe"} err="failed to get container status \"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe\": rpc error: code = NotFound desc = could not find container \"ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe\": container with ID starting with ea8285e25cd82798447a29e20fb6d46e48f917ca6d6d2e709105671830d811fe not found: ID does not exist"
Jul 29 12:04:22.997686 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.997668    2574 scope.go:117] "RemoveContainer" containerID="8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"
Jul 29 12:04:22.997905 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:04:22.997890    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce\": container with ID starting with 8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce not found: ID does not exist" containerID="8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"
Jul 29 12:04:22.997946 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:22.997910    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce"} err="failed to get container status \"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce\": rpc error: code = NotFound desc = could not find container \"8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce\": container with ID starting with 8863097300a70ba749ea90009ce5012e2a139407c3d553c76bf36b25c125f4ce not found: ID does not exist"
Jul 29 12:04:23.001560 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:23.001540    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:04:23.004373 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:23.004354    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-c98c474bb-tdxdt"]
Jul 29 12:04:24.342085 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:24.342055    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" path="/var/lib/kubelet/pods/1dcf1b3f-7f9e-48f6-85f1-b3277115352f/volumes"
Jul 29 12:04:36.524402 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524312    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524662    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524673    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524684    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="storage-initializer"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524693    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="storage-initializer"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524719    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524725    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524734    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="storage-initializer"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524739    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="storage-initializer"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524798    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51b094e-cecc-43ea-a352-0b38404b4590" containerName="main"
Jul 29 12:04:36.524853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.524806    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcf1b3f-7f9e-48f6-85f1-b3277115352f" containerName="main"
Jul 29 12:04:36.529291 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.529270    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.531056 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.531031    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 29 12:04:36.531250 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.531138    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-ntkdx\""
Jul 29 12:04:36.531559 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.531542    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 29 12:04:36.531619 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.531578    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 29 12:04:36.536394 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.536371    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:04:36.635948 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.635922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.636104 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.635977    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.636104 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.636033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk5l\" (UniqueName: \"kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.636104 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.636071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.636104 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.636095    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.636336 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.636111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.736945 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.736917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737095 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.736955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk5l\" (UniqueName: \"kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737095 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.736979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737095 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.736995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737095 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.737008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737095 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.737063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737428 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.737390    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737428 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.737412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.737513 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.737461    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.739534 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.739510    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.739666 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.739600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.744804 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.744784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk5l\" (UniqueName: \"kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l\") pod \"rolling-upgrade-test-kserve-74b79d7f77-qckmb\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.841885 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.841828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:36.974170 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:36.974146    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:04:36.976547 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:04:36.976521    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05494699_3335_4839_8d1f_434bc2775d69.slice/crio-e9d8a4812f4305f0ad206e79448247365f4f0702d85d0cb103152242a5a0a81c WatchSource:0}: Error finding container e9d8a4812f4305f0ad206e79448247365f4f0702d85d0cb103152242a5a0a81c: Status 404 returned error can't find the container with id e9d8a4812f4305f0ad206e79448247365f4f0702d85d0cb103152242a5a0a81c
Jul 29 12:04:37.053845 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:37.053812    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" event={"ID":"05494699-3335-4839-8d1f-434bc2775d69","Type":"ContainerStarted","Data":"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"}
Jul 29 12:04:37.053962 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:37.053872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" event={"ID":"05494699-3335-4839-8d1f-434bc2775d69","Type":"ContainerStarted","Data":"e9d8a4812f4305f0ad206e79448247365f4f0702d85d0cb103152242a5a0a81c"}
Jul 29 12:04:37.842261 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:37.842207    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:46.842620 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:46.842587    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:46.855038 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:46.855017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:04:46.870643 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:46.870600    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" podStartSLOduration=10.870589479 podStartE2EDuration="10.870589479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:04:37.068320686 +0000 UTC m=+5753.248702568" watchObservedRunningTime="2026-07-29 12:04:46.870589479 +0000 UTC m=+5763.050971364"
Jul 29 12:04:47.115340 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:04:47.115274    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:05:53.418063 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.418028    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:05:53.421549 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.421527    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.433278 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.433257    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:05:53.516687 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516655    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.516687 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516686    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.516871 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.516871 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.516871 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516749    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.516871 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.516811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xs2\" (UniqueName: \"kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617373 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xs2\" (UniqueName: \"kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617514 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617402    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617514 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617514 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617514 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617500    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617719 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617820 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617800    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617887 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617816    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.617964 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.617940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.619807 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.619784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.620053 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.620036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.623907 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.623884    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xs2\" (UniqueName: \"kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-7hh5s\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.731248 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.731165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:05:53.865704 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:53.865666    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:05:53.869246 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:05:53.869201    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872e943e_620b_4eda_8af2_c17fee291134.slice/crio-3446047401acea2657e3199246764575a75f5506aa744fb35bae3b114de5754f WatchSource:0}: Error finding container 3446047401acea2657e3199246764575a75f5506aa744fb35bae3b114de5754f: Status 404 returned error can't find the container with id 3446047401acea2657e3199246764575a75f5506aa744fb35bae3b114de5754f
Jul 29 12:05:54.452365 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:54.452330    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" event={"ID":"872e943e-620b-4eda-8af2-c17fee291134","Type":"ContainerStarted","Data":"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"}
Jul 29 12:05:54.452729 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:54.452382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" event={"ID":"872e943e-620b-4eda-8af2-c17fee291134","Type":"ContainerStarted","Data":"3446047401acea2657e3199246764575a75f5506aa744fb35bae3b114de5754f"}
Jul 29 12:05:54.470907 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:54.470852    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" podStartSLOduration=1.4708339590000001 podStartE2EDuration="1.470833959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:05:54.470148879 +0000 UTC m=+5830.650530759" watchObservedRunningTime="2026-07-29 12:05:54.470833959 +0000 UTC m=+5830.651215840"
Jul 29 12:05:54.732806 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:05:54.732711    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:03.732253 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:03.732193    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:03.744455 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:03.744429    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:04.514462 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:04.514428    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:04.557781 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:04.557751    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:06:04.567593 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:04.567554    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 12:06:04.567593 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:04.567593 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:04.567593 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3" cmd=["/bin/sleep","15"]
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:04.567611    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" message="" err=<
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:04.567633    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" podUID="05494699-3335-4839-8d1f-434bc2775d69" containerName="main" containerID="cri-o://460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"
Jul 29 12:06:04.567870 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:04.567674    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" podUID="05494699-3335-4839-8d1f-434bc2775d69" containerName="main" containerID="cri-o://460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3" gracePeriod=60
Jul 29 12:06:04.918762 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:04.918732    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:06:05.115530 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115424    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115530 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115481    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115530 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115505    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115544    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xk5l\" (UniqueName: \"kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115569    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115600    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir\") pod \"05494699-3335-4839-8d1f-434bc2775d69\" (UID: \"05494699-3335-4839-8d1f-434bc2775d69\") "
Jul 29 12:06:05.115789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115714    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home" (OuterVolumeSpecName: "home") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:05.115789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115730    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache" (OuterVolumeSpecName: "model-cache") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:05.116028 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115934    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.116028 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115947    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.116028 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.115952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:05.118042 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.118019    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l" (OuterVolumeSpecName: "kube-api-access-8xk5l") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "kube-api-access-8xk5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:06:05.118042 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.118027    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:06:05.118042 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.118031    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm" (OuterVolumeSpecName: "dshm") pod "05494699-3335-4839-8d1f-434bc2775d69" (UID: "05494699-3335-4839-8d1f-434bc2775d69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:05.216476 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.216452    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.216476 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.216472    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk5l\" (UniqueName: \"kubernetes.io/projected/05494699-3335-4839-8d1f-434bc2775d69-kube-api-access-8xk5l\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.216624 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.216482    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05494699-3335-4839-8d1f-434bc2775d69-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.216624 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.216491    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05494699-3335-4839-8d1f-434bc2775d69-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:05.509791 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.509705    2574 generic.go:356] "Generic (PLEG): container finished" podID="05494699-3335-4839-8d1f-434bc2775d69" containerID="460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3" exitCode=0
Jul 29 12:06:05.509791 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.509748    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"
Jul 29 12:06:05.510022 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.509785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" event={"ID":"05494699-3335-4839-8d1f-434bc2775d69","Type":"ContainerDied","Data":"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"}
Jul 29 12:06:05.510022 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.509839    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb" event={"ID":"05494699-3335-4839-8d1f-434bc2775d69","Type":"ContainerDied","Data":"e9d8a4812f4305f0ad206e79448247365f4f0702d85d0cb103152242a5a0a81c"}
Jul 29 12:06:05.510022 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.509860    2574 scope.go:117] "RemoveContainer" containerID="460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"
Jul 29 12:06:05.519558 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.519542    2574 scope.go:117] "RemoveContainer" containerID="460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"
Jul 29 12:06:05.519829 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:05.519812    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3\": container with ID starting with 460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3 not found: ID does not exist" containerID="460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"
Jul 29 12:06:05.519876 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.519840    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3"} err="failed to get container status \"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3\": rpc error: code = NotFound desc = could not find container \"460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3\": container with ID starting with 460e1c8835477c48c12c847b88c0f9afdc55eca6f7a3912a900bbffdd8d324d3 not found: ID does not exist"
Jul 29 12:06:05.529936 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.529914    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:06:05.535736 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:05.535716    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-qckmb"]
Jul 29 12:06:06.342347 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:06.342320    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05494699-3335-4839-8d1f-434bc2775d69" path="/var/lib/kubelet/pods/05494699-3335-4839-8d1f-434bc2775d69/volumes"
Jul 29 12:06:07.463711 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.463679    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:06:07.473934 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:07.473890    2574 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 29 12:06:07.473934 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:07.473934 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:07.473934 ip-10-0-133-86 kubenswrapper[2574]:  > containerID="04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b" cmd=["/bin/sleep","15"]
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.473954    2574 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" message="" err=<
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:  >
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:07.473972    2574 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:         , stdout: , stderr: , exit code -1
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main" containerID="cri-o://04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"
Jul 29 12:06:07.474160 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.474025    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main" containerID="cri-o://04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b" gracePeriod=60
Jul 29 12:06:07.503872 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.503841    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 29 12:06:07.820928 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.820907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:07.834921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.834892    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835037 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.834943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75xs2\" (UniqueName: \"kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835037 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.834983    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835159 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835053    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835159 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835159 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835109    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache\") pod \"872e943e-620b-4eda-8af2-c17fee291134\" (UID: \"872e943e-620b-4eda-8af2-c17fee291134\") "
Jul 29 12:06:07.835339 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835242    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:07.835399 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835346    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home" (OuterVolumeSpecName: "home") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:07.835448 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache" (OuterVolumeSpecName: "model-cache") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:07.835493 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835480    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-home\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:07.835530 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.835497    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-tmp-dir\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:07.837804 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.837765    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2" (OuterVolumeSpecName: "kube-api-access-75xs2") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "kube-api-access-75xs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:06:07.837913 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.837826    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm" (OuterVolumeSpecName: "dshm") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 12:06:07.838110 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.838084    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "872e943e-620b-4eda-8af2-c17fee291134" (UID: "872e943e-620b-4eda-8af2-c17fee291134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:06:07.936318 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.936298    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/872e943e-620b-4eda-8af2-c17fee291134-tls-certs\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:07.936318 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.936318    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75xs2\" (UniqueName: \"kubernetes.io/projected/872e943e-620b-4eda-8af2-c17fee291134-kube-api-access-75xs2\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:07.936318 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.936327    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-dshm\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:07.936500 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:07.936336    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/872e943e-620b-4eda-8af2-c17fee291134-model-cache\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:06:08.527717 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.527684    2574 generic.go:356] "Generic (PLEG): container finished" podID="872e943e-620b-4eda-8af2-c17fee291134" containerID="04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b" exitCode=0
Jul 29 12:06:08.528091 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.527727    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"
Jul 29 12:06:08.528091 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.527765    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" event={"ID":"872e943e-620b-4eda-8af2-c17fee291134","Type":"ContainerDied","Data":"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"}
Jul 29 12:06:08.528091 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.527811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s" event={"ID":"872e943e-620b-4eda-8af2-c17fee291134","Type":"ContainerDied","Data":"3446047401acea2657e3199246764575a75f5506aa744fb35bae3b114de5754f"}
Jul 29 12:06:08.528091 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.527833    2574 scope.go:117] "RemoveContainer" containerID="04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"
Jul 29 12:06:08.536868 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.536849    2574 scope.go:117] "RemoveContainer" containerID="04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"
Jul 29 12:06:08.537111 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:06:08.537091    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b\": container with ID starting with 04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b not found: ID does not exist" containerID="04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"
Jul 29 12:06:08.537161 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.537118    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b"} err="failed to get container status \"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b\": rpc error: code = NotFound desc = could not find container \"04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b\": container with ID starting with 04d56a394e79226f0075f6a4126f86060dee42dbaccfaca565157e5366c3713b not found: ID does not exist"
Jul 29 12:06:08.543633 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.543598    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:06:08.546407 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:08.546388    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-7hh5s"]
Jul 29 12:06:10.341327 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:10.341296    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872e943e-620b-4eda-8af2-c17fee291134" path="/var/lib/kubelet/pods/872e943e-620b-4eda-8af2-c17fee291134/volumes"
Jul 29 12:06:27.815487 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815453    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84977dd55d-zfng5"]
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815779    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05494699-3335-4839-8d1f-434bc2775d69" containerName="main"
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815788    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="05494699-3335-4839-8d1f-434bc2775d69" containerName="main"
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815796    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main"
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815801    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main"
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815852    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="05494699-3335-4839-8d1f-434bc2775d69" containerName="main"
Jul 29 12:06:27.815921 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.815859    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="872e943e-620b-4eda-8af2-c17fee291134" containerName="main"
Jul 29 12:06:27.818870 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.818849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.827283 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.827262    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84977dd55d-zfng5"]
Jul 29 12:06:27.883566 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.883541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9ww2\" (UniqueName: \"kubernetes.io/projected/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-kube-api-access-n9ww2\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.883669 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.883589    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-cert\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.984753 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.984728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9ww2\" (UniqueName: \"kubernetes.io/projected/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-kube-api-access-n9ww2\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.984853 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.984773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-cert\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.987331 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.987308    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-cert\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:27.992005 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:27.991976    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9ww2\" (UniqueName: \"kubernetes.io/projected/3e6f68da-0ace-42c7-916d-c6fd8fc5dae1-kube-api-access-n9ww2\") pod \"llmisvc-controller-manager-84977dd55d-zfng5\" (UID: \"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1\") " pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:28.128291 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:28.128240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:28.259705 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:28.259673    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84977dd55d-zfng5"]
Jul 29 12:06:28.262736 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:06:28.262709    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e6f68da_0ace_42c7_916d_c6fd8fc5dae1.slice/crio-05038c4689a83d3ffd3add3dbd90f4d5ec20ba8cee71107f9becc5316b65c6f4 WatchSource:0}: Error finding container 05038c4689a83d3ffd3add3dbd90f4d5ec20ba8cee71107f9becc5316b65c6f4: Status 404 returned error can't find the container with id 05038c4689a83d3ffd3add3dbd90f4d5ec20ba8cee71107f9becc5316b65c6f4
Jul 29 12:06:28.264176 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:28.264160    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 12:06:28.627649 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:28.627623    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5" event={"ID":"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1","Type":"ContainerStarted","Data":"05038c4689a83d3ffd3add3dbd90f4d5ec20ba8cee71107f9becc5316b65c6f4"}
Jul 29 12:06:29.634119 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:29.634079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5" event={"ID":"3e6f68da-0ace-42c7-916d-c6fd8fc5dae1","Type":"ContainerStarted","Data":"84d1d354b3b730817c9af6671f865c4e17f6ae3347695ecb4d0f6229736f7bca"}
Jul 29 12:06:29.634562 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:29.634179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:06:29.648951 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:06:29.648903    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5" podStartSLOduration=1.99344506 podStartE2EDuration="2.648889465s" totalImagesPullingTime="655.444405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:06:29.647369978 +0000 UTC m=+5865.827751859" watchObservedRunningTime="2026-07-29 12:06:29.648889465 +0000 UTC m=+5865.829271344"
Jul 29 12:07:00.640651 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:00.640619    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84977dd55d-zfng5"
Jul 29 12:07:00.678570 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:00.678537    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 12:07:00.678822 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:00.678774    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" podUID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" containerName="manager" containerID="cri-o://5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb" gracePeriod=30
Jul 29 12:07:03.029847 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.029820    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 12:07:03.140356 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.140281    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8jnw\" (UniqueName: \"kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw\") pod \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") "
Jul 29 12:07:03.140356 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.140353    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") pod \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\" (UID: \"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4\") "
Jul 29 12:07:03.142514 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.142488    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert" (OuterVolumeSpecName: "cert") pod "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" (UID: "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 12:07:03.142632 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.142517    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw" (OuterVolumeSpecName: "kube-api-access-q8jnw") pod "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" (UID: "e0741931-2ec6-4c8f-bf3b-de4f6441f8c4"). InnerVolumeSpecName "kube-api-access-q8jnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 12:07:03.241733 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.241703    2574 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-cert\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:07:03.241733 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.241726    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jnw\" (UniqueName: \"kubernetes.io/projected/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4-kube-api-access-q8jnw\") on node \"ip-10-0-133-86.ec2.internal\" DevicePath \"\""
Jul 29 12:07:03.803188 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.803156    2574 generic.go:356] "Generic (PLEG): container finished" podID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" containerID="5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb" exitCode=0
Jul 29 12:07:03.803358 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.803203    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc"
Jul 29 12:07:03.803358 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.803243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" event={"ID":"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4","Type":"ContainerDied","Data":"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"}
Jul 29 12:07:03.803358 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.803299    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-67c47db88-fw5qc" event={"ID":"e0741931-2ec6-4c8f-bf3b-de4f6441f8c4","Type":"ContainerDied","Data":"9c893595dc5ab0eae916ecc8f2600af17a7772054ddda7415dd1e619cf902273"}
Jul 29 12:07:03.803358 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.803322    2574 scope.go:117] "RemoveContainer" containerID="5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"
Jul 29 12:07:03.816565 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.816549    2574 scope.go:117] "RemoveContainer" containerID="5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"
Jul 29 12:07:03.816807 ip-10-0-133-86 kubenswrapper[2574]: E0729 12:07:03.816788    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb\": container with ID starting with 5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb not found: ID does not exist" containerID="5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"
Jul 29 12:07:03.816857 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.816816    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb"} err="failed to get container status \"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb\": rpc error: code = NotFound desc = could not find container \"5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb\": container with ID starting with 5ac5eddf8448c85a184bd3ec9beba3dbdac2dae07619b556cf64851f61bc5dbb not found: ID does not exist"
Jul 29 12:07:03.822721 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.822701    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 12:07:03.824711 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:03.824692    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-67c47db88-fw5qc"]
Jul 29 12:07:04.342854 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:07:04.342827    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" path="/var/lib/kubelet/pods/e0741931-2ec6-4c8f-bf3b-de4f6441f8c4/volumes"
Jul 29 12:08:46.587706 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:08:46.587616    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 12:08:46.622060 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:08:46.622041    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 12:11:57.031477 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:11:57.031437    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"]
Jul 29 12:11:57.036019 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:11:57.035996    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-hcdhx"]
Jul 29 12:11:58.341909 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:11:58.341874    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93495d2b-b97d-498f-b0ee-8a27962fd428" path="/var/lib/kubelet/pods/93495d2b-b97d-498f-b0ee-8a27962fd428/volumes"
Jul 29 12:12:26.423545 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:26.423458    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:26.466314 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:26.466290    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:26.818474 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:26.818441    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:27.720815 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:27.720770    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:27.892479 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:27.892456    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:28.334655 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:28.334626    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:29.217779 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:29.217742    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:29.267637 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:29.267614    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:29.580721 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:29.580694    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:30.443913 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:30.443881    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:30.491816 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:30.491795    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:30.810661 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:30.810640    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:31.660496 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:31.660461    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:31.707014 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:31.706992    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:32.025812 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:32.025788    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:32.870129 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:32.870096    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:32.921524 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:32.921499    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:33.244659 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:33.244594    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:34.114185 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:34.114158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:34.162040 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:34.162015    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:34.487441 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:34.487363    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:35.328082 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:35.328052    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:35.377023 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:35.376997    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:35.693043 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:35.692973    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:36.545345 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:36.545318    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:36.595664 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:36.595642    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:36.913613 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:36.913546    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:37.776394 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:37.776363    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:37.827879 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:37.827853    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:38.145644 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:38.145573    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:38.996020 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:38.995990    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:39.044211 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:39.044189    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:39.369079 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:39.369053    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:40.252971 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:40.252938    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:40.304765 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:40.304743    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:40.621407 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:40.621379    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:41.460893 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:41.460863    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:41.506385 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:41.506360    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:41.820166 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:41.820143    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:42.661026 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:42.660995    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:42.706305 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:42.706286    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:43.027433 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:43.027400    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:43.879508 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:43.879480    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:43.933992 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:43.933970    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:44.256745 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:44.256676    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:45.109987 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:45.109951    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:45.155890 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:45.155865    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:45.202886 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:45.202857    2574 scope.go:117] "RemoveContainer" containerID="a5857a1d7135efabe8202a139f1891b033a56d3d3d73fd5db6bd884197404195"
Jul 29 12:12:45.496147 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:45.496085    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:46.358726 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:46.358694    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:46.406590 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:46.406564    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:46.723087 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:46.723007    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:47.588459 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:47.588429    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:47.641293 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:47.641269    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:47.973883 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:47.973811    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:48.833237 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:48.833184    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:48.880204 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:48.880179    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:49.201881 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:49.201814    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:50.072119 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:50.072082    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:50.117599 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:50.117572    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:50.436906 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:50.436829    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:51.303302 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:51.303270    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:51.354192 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:51.354171    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:51.673134 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:51.673065    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:52.531187 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:52.531155    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:52.579001 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:52.578969    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:52.909176 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:52.909099    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:53.758950 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:53.758922    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:53.808073 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:53.808052    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:54.124500 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:54.124474    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:54.955903 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:54.955876    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:55.007148 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:55.007127    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:55.330075 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:55.330049    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:56.189198 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:56.189165    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:12:56.235515 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:56.235491    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:12:56.552067 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:56.552045    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:12:58.668603 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.668569    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjmjm/must-gather-cwdzq"]
Jul 29 12:12:58.669045 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.668922    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" containerName="manager"
Jul 29 12:12:58.669045 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.668934    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" containerName="manager"
Jul 29 12:12:58.669045 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.668987    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0741931-2ec6-4c8f-bf3b-de4f6441f8c4" containerName="manager"
Jul 29 12:12:58.673467 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.673440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.675238 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.675202    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jjmjm\"/\"default-dockercfg-zcqgd\""
Jul 29 12:12:58.675334 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.675200    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"openshift-service-ca.crt\""
Jul 29 12:12:58.675642 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.675625    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jjmjm\"/\"kube-root-ca.crt\""
Jul 29 12:12:58.680728 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.680408    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/must-gather-cwdzq"]
Jul 29 12:12:58.721120 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.721096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cb307fa-e464-4ece-8d1a-f048e4e30351-must-gather-output\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.721248 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.721136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mzh\" (UniqueName: \"kubernetes.io/projected/5cb307fa-e464-4ece-8d1a-f048e4e30351-kube-api-access-q8mzh\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.821990 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.821962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cb307fa-e464-4ece-8d1a-f048e4e30351-must-gather-output\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.822086 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.822001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mzh\" (UniqueName: \"kubernetes.io/projected/5cb307fa-e464-4ece-8d1a-f048e4e30351-kube-api-access-q8mzh\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.822299 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.822283    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5cb307fa-e464-4ece-8d1a-f048e4e30351-must-gather-output\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.828940 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.828922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mzh\" (UniqueName: \"kubernetes.io/projected/5cb307fa-e464-4ece-8d1a-f048e4e30351-kube-api-access-q8mzh\") pod \"must-gather-cwdzq\" (UID: \"5cb307fa-e464-4ece-8d1a-f048e4e30351\") " pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:58.983702 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:58.983650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/must-gather-cwdzq"
Jul 29 12:12:59.113561 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:59.113515    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/must-gather-cwdzq"]
Jul 29 12:12:59.115740 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:12:59.115712    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb307fa_e464_4ece_8d1a_f048e4e30351.slice/crio-d1df59ade2c7a32094ab11766bf7e9570abef5335676261ad58717a4cb1f49fa WatchSource:0}: Error finding container d1df59ade2c7a32094ab11766bf7e9570abef5335676261ad58717a4cb1f49fa: Status 404 returned error can't find the container with id d1df59ade2c7a32094ab11766bf7e9570abef5335676261ad58717a4cb1f49fa
Jul 29 12:12:59.117723 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:59.117707    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 12:12:59.564200 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:12:59.564170    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/must-gather-cwdzq" event={"ID":"5cb307fa-e464-4ece-8d1a-f048e4e30351","Type":"ContainerStarted","Data":"d1df59ade2c7a32094ab11766bf7e9570abef5335676261ad58717a4cb1f49fa"}
Jul 29 12:13:00.574541 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:00.574501    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/must-gather-cwdzq" event={"ID":"5cb307fa-e464-4ece-8d1a-f048e4e30351","Type":"ContainerStarted","Data":"595aa83dd9595026082d70ef35e5d268c659dc8a541375d6fd0c48036ed5554a"}
Jul 29 12:13:00.575024 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:00.574557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/must-gather-cwdzq" event={"ID":"5cb307fa-e464-4ece-8d1a-f048e4e30351","Type":"ContainerStarted","Data":"442057b99209995a60a6db1e99d0302dd81009be778ee37e832611549b2efe53"}
Jul 29 12:13:00.588734 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:00.588686    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjmjm/must-gather-cwdzq" podStartSLOduration=1.691935202 podStartE2EDuration="2.588670507s" totalImagesPullingTime="896.735305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 12:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:13:00.587983619 +0000 UTC m=+6256.768365516" watchObservedRunningTime="2026-07-29 12:13:00.588670507 +0000 UTC m=+6256.769052386"
Jul 29 12:13:01.547576 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:01.547539    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dsndc_78f23bd5-32c5-4c6e-a8b1-a72745649bcd/global-pull-secret-syncer/0.log"
Jul 29 12:13:01.748174 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:01.748129    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dsdrb_c3043fc9-285c-4a12-88c9-3566d826375a/konnectivity-agent/0.log"
Jul 29 12:13:01.866851 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:01.866766    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-86.ec2.internal_0d1203cd46ad847a51fd730c0dc9ddcf/haproxy/0.log"
Jul 29 12:13:05.718019 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:05.717985    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-5j7sb_543a1ddc-23da-4669-9cb2-4160f91d3982/authorino/0.log"
Jul 29 12:13:05.809666 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:05.809632    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-gwjp6_0ac9ee07-648c-493f-81e2-585817064060/kuadrant-console-plugin/0.log"
Jul 29 12:13:06.150556 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:06.150532    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c9rqr_d8099bc5-4964-415e-bd97-b807216c74ab/limitador/0.log"
Jul 29 12:13:08.284456 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.284425    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbrk_49ae2060-41df-42dd-ab80-7787fa700f8a/node-exporter/0.log"
Jul 29 12:13:08.307593 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.307547    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbrk_49ae2060-41df-42dd-ab80-7787fa700f8a/kube-rbac-proxy/0.log"
Jul 29 12:13:08.333290 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.333180    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rfbrk_49ae2060-41df-42dd-ab80-7787fa700f8a/init-textfile/0.log"
Jul 29 12:13:08.686832 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.686805    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-flfln_f855927d-a2bb-4ebf-b294-a7ffb58788d5/prometheus-operator/0.log"
Jul 29 12:13:08.708675 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.708646    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-flfln_f855927d-a2bb-4ebf-b294-a7ffb58788d5/kube-rbac-proxy/0.log"
Jul 29 12:13:08.740759 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.740714    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-2dqsh_916a79bd-ceb2-437c-bae4-03eaa853c62f/prometheus-operator-admission-webhook/0.log"
Jul 29 12:13:08.791644 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.791616    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f995bbb7b-q4trf_795c50eb-aa9a-46eb-9c7b-b8efe347a2ec/telemeter-client/0.log"
Jul 29 12:13:08.819912 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.819886    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f995bbb7b-q4trf_795c50eb-aa9a-46eb-9c7b-b8efe347a2ec/reload/0.log"
Jul 29 12:13:08.843997 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:08.843963    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f995bbb7b-q4trf_795c50eb-aa9a-46eb-9c7b-b8efe347a2ec/kube-rbac-proxy/0.log"
Jul 29 12:13:10.515361 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:10.515330    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-cdrjg_825761a1-4047-4058-b59b-24871c8e45e5/networking-console-plugin/0.log"
Jul 29 12:13:11.133414 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:11.133382    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/1.log"
Jul 29 12:13:11.138918 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:11.138885    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bmxdx_20188e15-550f-46cd-b99b-a5bf59756d5b/console-operator/2.log"
Jul 29 12:13:11.580374 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:11.580353    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b9cc94695-plmjd_815906ea-4f97-48bd-a50d-0aceee338f9e/console/0.log"
Jul 29 12:13:11.624789 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:11.624760    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-6kv2b_52732dce-c95d-4b8a-b2f2-463b0102037b/download-server/0.log"
Jul 29 12:13:12.284804 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.284764    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"]
Jul 29 12:13:12.289813 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.289788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.295151 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.295123    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"]
Jul 29 12:13:12.378452 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.378421    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-podres\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.378643 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.378482    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64s5\" (UniqueName: \"kubernetes.io/projected/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-kube-api-access-m64s5\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.378643 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.378533    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-proc\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.378643 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.378558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-lib-modules\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.378846 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.378659    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-sys\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479196 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-proc\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479408 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-lib-modules\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479408 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-sys\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479408 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479326    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-proc\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479408 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-podres\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479408 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64s5\" (UniqueName: \"kubernetes.io/projected/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-kube-api-access-m64s5\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479597 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-lib-modules\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479733 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479701    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-podres\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.479814 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.479730    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-sys\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.486694 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.486669    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64s5\" (UniqueName: \"kubernetes.io/projected/8b7cbebb-6575-43a8-8ebf-a794a9faa88b-kube-api-access-m64s5\") pod \"perf-node-gather-daemonset-xthg4\" (UID: \"8b7cbebb-6575-43a8-8ebf-a794a9faa88b\") " pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.612763 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.612673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:12.800393 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:12.800352    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"]
Jul 29 12:13:12.805490 ip-10-0-133-86 kubenswrapper[2574]: W0729 12:13:12.805461    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8b7cbebb_6575_43a8_8ebf_a794a9faa88b.slice/crio-fe661eb9dd4e442e8844192ad0695869459c424dc25ff9bd0fa0302509c4ece3 WatchSource:0}: Error finding container fe661eb9dd4e442e8844192ad0695869459c424dc25ff9bd0fa0302509c4ece3: Status 404 returned error can't find the container with id fe661eb9dd4e442e8844192ad0695869459c424dc25ff9bd0fa0302509c4ece3
Jul 29 12:13:13.299830 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.299802    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2cth_df56f12f-8167-4d7b-8863-7a21eeba4d97/dns/0.log"
Jul 29 12:13:13.320444 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.320419    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2cth_df56f12f-8167-4d7b-8863-7a21eeba4d97/kube-rbac-proxy/0.log"
Jul 29 12:13:13.392007 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.391972    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gl58w_2818e4ae-44ec-460b-a866-86b544e12a30/dns-node-resolver/0.log"
Jul 29 12:13:13.665888 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.665798    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4" event={"ID":"8b7cbebb-6575-43a8-8ebf-a794a9faa88b","Type":"ContainerStarted","Data":"21309556adb0113239b575ee37458755748985da92b5cd8108c07bfe28c0e7f7"}
Jul 29 12:13:13.665888 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.665859    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:13.665888 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.665882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4" event={"ID":"8b7cbebb-6575-43a8-8ebf-a794a9faa88b","Type":"ContainerStarted","Data":"fe661eb9dd4e442e8844192ad0695869459c424dc25ff9bd0fa0302509c4ece3"}
Jul 29 12:13:13.681079 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.679309    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4" podStartSLOduration=1.679289996 podStartE2EDuration="1.679289996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 12:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 12:13:13.67856795 +0000 UTC m=+6269.858949832" watchObservedRunningTime="2026-07-29 12:13:13.679289996 +0000 UTC m=+6269.859671876"
Jul 29 12:13:13.976508 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:13.976432    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6nvz2_a6c58d40-a2da-4b45-aa50-091fe6d34cca/node-ca/0.log"
Jul 29 12:13:15.991758 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:15.991726    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8tqhw_e0cd63fd-0295-493e-a581-d3af51c97345/serve-healthcheck-canary/0.log"
Jul 29 12:13:16.604499 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:16.604452    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6pkk_5e233220-65a3-4899-b378-8fbccc025d1a/kube-rbac-proxy/0.log"
Jul 29 12:13:16.625086 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:16.625060    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6pkk_5e233220-65a3-4899-b378-8fbccc025d1a/exporter/0.log"
Jul 29 12:13:16.648956 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:16.648931    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-q6pkk_5e233220-65a3-4899-b378-8fbccc025d1a/extractor/0.log"
Jul 29 12:13:19.522286 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:19.522257    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c847d9c7c-9t4vb_27cbc149-314b-4b4d-acd6-52452642acd5/manager/0.log"
Jul 29 12:13:19.683852 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:19.683825    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jjmjm/perf-node-gather-daemonset-xthg4"
Jul 29 12:13:20.203536 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:20.203501    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84977dd55d-zfng5_3e6f68da-0ace-42c7-916d-c6fd8fc5dae1/manager/0.log"
Jul 29 12:13:21.572326 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:21.572288    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bg8rw_dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e/s3-init/0.log"
Jul 29 12:13:21.603551 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:21.603484    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-bg8rw_dd3919b7-dcc8-43b8-b2c1-94c6fcb8653e/download-hf-model/0.log"
Jul 29 12:13:27.014207 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:27.014161    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb_95ec880c-ff89-4c96-91d7-a390b557c278/kube-storage-version-migrator-operator/1.log"
Jul 29 12:13:27.015687 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:27.015660    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wlmvb_95ec880c-ff89-4c96-91d7-a390b557c278/kube-storage-version-migrator-operator/0.log"
Jul 29 12:13:28.121770 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.121736    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9ncwb_085098cb-f1e3-4b6a-ac1d-ef7528e75bcb/kube-multus/0.log"
Jul 29 12:13:28.312083 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.312031    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/kube-multus-additional-cni-plugins/0.log"
Jul 29 12:13:28.349364 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.349338    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/egress-router-binary-copy/0.log"
Jul 29 12:13:28.370254 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.370229    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/cni-plugins/0.log"
Jul 29 12:13:28.391364 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.391305    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/bond-cni-plugin/0.log"
Jul 29 12:13:28.412570 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.412550    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/routeoverride-cni/0.log"
Jul 29 12:13:28.433741 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.433720    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/whereabouts-cni-bincopy/0.log"
Jul 29 12:13:28.456447 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.456426    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lk64z_5442ddcf-c235-46d4-a3bb-faace9f9f281/whereabouts-cni/0.log"
Jul 29 12:13:28.702109 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.702036    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5fm96_2df1e671-4dab-4eed-b983-326d3fc53b9c/network-metrics-daemon/0.log"
Jul 29 12:13:28.722357 ip-10-0-133-86 kubenswrapper[2574]: I0729 12:13:28.722333    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5fm96_2df1e671-4dab-4eed-b983-326d3fc53b9c/kube-rbac-proxy/0.log"